Using a regular expression, in PyCharm:
Search:
"\{\{ MEDIA_URL \}\}\/(.*?)"
Replace with:
"{% static "$1" %}"
This also is a note that that in PyCharm, the way to reference capture groups in the replace expression, is the dollar sign.
Django: Moving from MEDIA_URL to the {% static %} tag
Posted in Uncategorized
Benchmark Dump
The hardware is a Intel Xeon E5506 4x 2.33GHz, running the free version of ESXi. All ESXi disks where set to “Thick-Provision Lazy Zeroed”. The ext4 filesystem was used for all tests.
Virtual Machine running CentOS 6.3
./nbench
BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)
TEST : Iterations/sec. : Old Index : New Index
: : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT : 1008.8 : 25.87 : 8.50
STRING SORT : 208.96 : 93.37 : 14.45
BITFIELD : 4.1418e+08 : 71.05 : 14.84
FP EMULATION : 215.4 : 103.36 : 23.85
FOURIER : 24877 : 28.29 : 15.89
ASSIGNMENT : 27.847 : 105.96 : 27.48
IDEA : 6380 : 97.58 : 28.97
HUFFMAN : 2180.1 : 60.46 : 19.31
NEURAL NET : 53.557 : 86.04 : 36.19
LU DECOMPOSITION : 1491.3 : 77.26 : 55.79
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX : 73.038
FLOATING-POINT INDEX: 57.290
Baseline (MSDOS*) : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU : GenuineIntel Intel(R) Xeon(R) CPU L5506 @ 2.13GHz 2133MHz
L2 Cache : 4096 KB
OS : Linux 2.6.32-042stab068.8
C compiler : gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)
libc : libc-2.12.so
MEMORY INDEX : 18.064
INTEGER INDEX : 18.348
FLOATING-POINT INDEX: 31.775
Baseline (LINUX) : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.
Ubuntu Precise OpenVZ VE, 3GB RAM, CentOS 6 Host with LVM, 15.000 U/min SAS, One ESXi Snapshot, No VMware Tools
bonnie++ -n 150 -s 10g
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
olive 10G 651 99 80597 17 76153 25 2435 92 3783592 99 1394 346
Latency 14955us 2442ms 2269ms 101ms 1363us 23478us
Version 1.96 ------Sequential Create------ --------Random Create--------
olive -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
150 38559 62 +++++ +++ 7438 11 34394 51 +++++ +++ 5209 8
Latency 1356ms 555us 2694ms 2064ms 75042us 2427ms
1.96,1.96,olive,1,1356623492,10G,,651,99,80597,17,76153,25,2435,92,3783592,99,1394,346,150,,,,,38559,62,+++++,+++,7438,11,34394,51,+++++,+++,5209,8,14955us,2442ms,2269ms,101ms,1363us,23478us,1356ms,555us,2694ms,2064ms,75042us,2427ms
CentOS 6, 7200 U/min, No LVM, No Snapshots
No VMware Tools:
bonnie++ -s 8g -n 256
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
olive.dpool.org 8G 511 84 13799 1 57858 10 2914 98 117598 13 704.6 19
Latency 16796us 1943ms 992ms 12069us 34219us 130ms
Version 1.96 ------Sequential Create------ --------Random Create--------
olive.dpool.org -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
256 43041 60 +++++ +++ 2879 3 49716 67 +++++ +++ 1671 2
Latency 275ms 459us 5313ms 287ms 58us 5956ms
1.96,1.96,olive.dpool.org,1,1356642593,8G,,511,84,13799,1,57858,10,2914,98,117598,13,704.6,19,256,,,,,43041,60,+++++,+++,2879,3,49716,67,+++++,+++,1671,2,16796us,1943ms,992ms,12069us,34219us,130ms,275ms,459us,5313ms,287ms,58us,5956ms
With VMware Tools:
bonnie++ -s 8g -n 256
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
olive.dpool.org 8G 523 87 119178 14 56202 10 2586 98 116543 14 719.9 18
Latency 17037us 328ms 1052ms 7225us 34158us 89850us
Version 1.96 ------Sequential Create------ --------Random Create--------
olive.dpool.org -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
256 47724 67 +++++ +++ 2926 3 47189 69 +++++ +++ 1631 2
Latency 320ms 705us 5330ms 376ms 92us 7095ms
1.96,1.96,olive.dpool.org,1,1356647676,8G,,523,87,119178,14,56202,10,2586,98,116543,14,719.9,18,256,,,,,47724,67,+++++,+++,2926,3,47189,69,+++++,+++,1631,2,17037us,328ms,1052ms,7225us,34158us,89850us,320ms,705us,5330ms,376ms,92us,7095ms
Repeat:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
olive.dpool.org 8G 674 98 29959 3 51369 9 2819 98 116235 14 696.1 18
Latency 15236us 2303ms 971ms 13305us 57728us 146ms
Version 1.96 ------Sequential Create------ --------Random Create--------
olive.dpool.org -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
256 48954 68 +++++ +++ 2935 3 47420 64 +++++ +++ 1673 2
Latency 302ms 539us 5202ms 291ms 79us 6361ms
1.96,1.96,olive.dpool.org,1,1356646910,8G,,674,98,29959,3,51369,9,2819,98,116235,14,696.1,18,256,,,,,48954,68,+++++,+++,2935,3,47420,64,+++++,+++,1673,2,15236us,2303ms,971ms,13305us,57728us,146ms,302ms,539us,5202ms,291ms,79us,6361ms
Posted in Uncategorized
Name generators
- Suggests a mostly fixed set of variations.
- Very powerful, many variation modes.
- Not necessarily tailored for name suggestions, but offers a surprisingly diverse selection.
- Lets you setup combinations manually, check domains for availability.
Approaches
- Combinations: PayPal, PhotoShop, SourceForge
- Onomatopoeic: Twitter
- Associative/Personative: Dr. Pepper, Galileo
- Prefixes like go-, one-, or Suffixes like -ling, -ler, -let, -ly, -ster
Posted in Uncategorized
Trading Tools
- Apparently only for Forex.
- Can possibly do the things I need, but seems just too complicated.
- Works with Flatex
- Open Source
- Not that great.
- Seems popular, but has an ugly UI
- I haven’t tried it.
- Maybe closest to what I am looking for / Seems mostly accessible.
Posted in Uncategorized
Notes on scanning medium format 120 films
- I’m using a Nikon 9000.
- I want to use a RAW workflow.
The biggest PIA is frame detection. I’m scanning old B/W films, and the frame spacing is highly irregular.
- VueScan cannot auto-detect the frames. It allows you to provide a manual frame spacing, but with irregular spacing, you need to measure each strip manually, and in same cases, there is no valid value. I’m not sure what VueScan does if no frame spacing is configured (if there is an auto-detection, or if it is just assuming a fixed value, but at any rate, it doesn’t work). Given the right frame spacing though, VueScan successfully crops out the actual image from each frame scan.
- SilverFast, apart from having one of the worst UIs ever created and an irredeemable pricing model, also doesn’t auto-detect the frames. Though thankfully, it does have a UI tool to adjust the frame borders via repeated mouse clicks after an index scan. However, once done, it isn’t able to the crop box of the actual image within each frame. For some mysterious reason, the “find frames” menu item offers alternating choices like “6×4.5″ or “35mm”, but never something useful, like 6×6.
- NikonScan picks out the images perfectly. Alas, I don’t like it’s RAW handling.
So here’s what I ended up doing: I’m using VueScan, and I’m specifying the frame padding manually, but I’ve written a small tool to calculate the proper frame spacing value. Now, I just need to measure the spacings on a 3-frame strip (and the frame offset at the beginning), and it’ll give me a value that works for that strip. Still bothersome, but manageable.
Other settings I am using:
- Disabled Fine Scan and Multi exposure. I can’t see a difference.
- I’m scanning in color mode for now, just to be save.
Scanned film is archived in negative sleeves. I’ve put together another little program that helps me rename the files output by Vuescan so their filename indicates the sheet and strip number where the original can be found (say, 0044.3.2.jpg).
Interesting Links:
Posted in Uncategorized
Wireframing Tools
Posted in Uncategorized
.NET Twain Libraries
- At $325 more reasonable, and also seems to include source.
- Has a pretty nice custom configuration dialog, showing only the supported capabilities in a sort of inspector view.
- Sample apps are lacking a bit.
- Cheap, starting at 130 EUR.
- Source not available.
- Has extensive examples, even a specific demo for extended image info!
Posted in Uncategorized
On Nate Silver
He got lucky? Sigh. He determined the distribution of probabilities and we placed a bet for him, in the currency of his reputation.
The credit should go to Nate for being graceful under the fire from a hostile campaign and a self-righteous punditocracy. Credit to the Times for giving a talented, disciplined, genuinely decent stathead a national platform and establishment credibility. Credit to the organizations doing the state polls which Nate aggregated, and to the various levels of transparency and consistency which allowed Nate to usefully judge their merits. Credit to Major League Baseball, Billy Beane, Daniel Okrent, Bill James, and Michael Lewis for inspiring, influencing, and popularizing his methods and manners. Credit to everyone who ever used clear-eyed statistical analysis to win their fantasy league. Credit to the universe, for having a reliable tendency to exhibit scalar self-similarities.
And credit to us, for caring enough to pay attention.
Posted in Uncategorized
Going paperless
I am currently attempting to move to a paperless office. Here’s my notes investigating the technical side.
Goals
- This needs to be simple. Currently, filing a sheet of paper requires me to fetch the folder, punch the holes, then put it in the right register, finally put the folder back in the drawer. For the digital version, at the most, I want to place some sheets of paper into the scanner, press a button, be queried for a title and maybe some tags, then click Save and be done.
- The metadata I create must not be locked into some proprietary application. That means that any titles/keywords must be included with the PDF file. Ideally, any folder hierarchy would also be reflected in the filesystem, though obviously this is though if you also want to support one document being in multiple folders.
The scanner
Basically, the ScanSanp S1500 seems to be what everyone is using. In the same price segment, there is also the Canon DR-C125.
I chose the Canon because it is a bit smaller. Also because it has TWAIN. Though as it turns out, basically every program I tested supports the ScanSnap anyway, while some do not support TWAIN, so go figure.
I’m reasonably happy with the Canon. Pages are never scanned 100% without skew, but I assume that’s to be expected (most scan tools have deskew functionality). The ability to scan thick and longer-than-usual sheets of paper already came in useful. The optional ability to eject paper onto the surface in front of the scanner is troublesome; new sheets keep pushing the old ones away, and sometimes even end up below earlier ones.
A tool to scan and OCR
Nearly every document manager I tried supports taking documents directly from the scanner. Most apply some version of OCR while they’re at it. Many use the not-so-great Tesseract engine. None really do what I want.
When I’m digitizing my archive, and I put 30 sheets of paper in the scanner, some of which belong to the same document, I need an UI that allows me to quickly group those sheets together. An app generally might give you the option of either saving each scanned page separately, or to merge everything into one document.
The one exception is Abbyy FineReader 11, which does allow this sort of splitting off of scanned pages into new documents, with an interface even that is pretty close to what I’m a looking for, relatively speaking, but then opens a new window for each document, and requires you to OCR and save each one manually. It also likes to crash when doing this.
I didn’t have a close look at the OCR results of most of the apps I tried, but I did compare Capture OnTouch, the tool that ships with the Canon scanner, with Abbyy FineReader 11, and the latter is clearly superior. CaptureOnTouch doesn’t even recognize German umlauts.
I gather fron online sources that the OCR in Acrobat X is not supposed to be as good as Abbyy either. I did also not try OmniPage.
A tool to tag/organize/view
I tried a huge number of those, for Windows and OS X, mostly looking for one that a) integrates easy batch scanning, as described above, and b) isn’t a metadata blackbox. I’m not a happy camper.
Obviously, storing meta data is a hard problem.
- You can store it in the file, but are limited by formats; and changing the metadata means the file needs to be changed (which is potentially dangerous, impacts backups etc).
- You can store it via xattr or alternate streams, but it’s easy to lose these across filesytems.
- You can store it in a boxcar files, but they are also easy to lose, and cause clutter.
- Store them in an external database, and you are locked into the particular tool you are using.
Also, having a document in multiple categories cannot be easily mapped into the filesystem.
That said, here’s a quick rundown of the apps I do have tried, and the reason for dismissing them:
- Blackbox
- Stores metadata in xattr; neat, but not cross platform. I’m not trusting tools to not loose this data over a period of decades.
- Does support TWAIN scanners.
- Maybe the best of the bunch. *Very close*.
- Available on Windows, though that version isn’t quite as nice.
- Unfornately, does not support storing files outside it’s library package. This also means any collections I create are in a blackbox.
- Does OCR (using Tesseract), but doesn’t store it in the pdf itself, so this won’t do any harm to an OCR layer already there.
- Seems like a good tool, but also a total lock in.
- Simple and nice, but not a lot of functionality (not even PDF preview), basically just a list.
- Strange system of account / account numbers
- Quite heavy, didn’t really leave an impression.
- Uses OpenMeta for tags (xattr).
- But apparently cannot write them to the PDF file.
- Can work with PDFs that aren’t copied into it’s library package.
- But has it’s on folder/groups system which does not represent the filesystem.
- Very close.
- Mostly a black box, but allows to link an external folder structure (using the Index menu item, I didn’t get it at first).
- This allows you to use the app for search, but metadata changes made are not synced to the files.
- It doesn’t seem to show/search metadata from the actual PDF, but I may be mistaken here.
Other tools I tried: Papers (quite nice, but too research tailored), Medeley (also totally research tailored).
Resolution
At this point, I’ve decided to put together a small utility for myself to sort scanned pages into documents, and send the whole thing through OCR. I’m then going to store keywords directly in the PDF files, save those in a relatively flat folder structure, and use Windows search / iFilter to access them, forgoing a specialized GUI.
Posted in Uncategorized
Thoughts on Mass Effect 3 Extended Cut
Ultimately, I didn’t care for it.
No doubt the original endings, in terms of visual presentation, seemed a bit cheap, especially in the way they were essentially copies. (Though guess what, the new endings are still essentially copies). Still, some additional/extended scenes I am not necessarily unappreciated of.
But the changes make the ending boring and uninspired. Big fight, you win, all is good. The problem, I believe, is that the endings are not just visually similar. They are the same thing in all the things that matter (Destroy and Control most of all); the difference is academic and abstract. Shepard is (mostly) dead, everyone else is (generally) happy, and the Galaxy is saved. Tight and neat.
The old ending was itself somewhat abstract. It wasn’t as clear on what happened with the relays. Maybe earth would indeed have become the showplace of an another apocalyptic fight for sustenance. Maybe it was all intoxication. The ending, in its open-endedness, carried on the appropriate amount of depressing bleakness that the story was going for up to this point.
Because the details of what happened weren’t clear, the ending used to be a true Sophie’s Choice. Now, the additional explanations show your choice to be irrelevant.
Also, I’m not going to mention the ridiculous attempt at closing story holes that is the evac to the Normandy scene.
There’s also another article I enjoined, and my thoughts on the original ending.
Posted in Uncategorized