Random notes and comments about YAPC::NA in Madison, WI
ack 2.0
I uploaded ack 2.00alpha01 to the CPAN.
All that week, Rob Hoelz did a ton of work, and Jerry Gay was invaluable in helping us work through some configuration issues. Then, out of nowhere, Ryan Olson swoops in to close some sticky issues in the GitHub queue. I love conferences for bringing people together to get things done.
Finally, on Thursday night at the Bad Movie BOF I hacked away on the final few tickets while watching “Computer Beach Party (1987)”. Halfway through MST3K’s take on “Catalina Caper (1967)”, I made the alpha release. If that’s not heaven, I don’t know what is.
Mojolicious
Glen Hinkle
Mojolicous looks really cool. Glen called it a “full web framework, not partial,” although I’m not sure what would count as a partial framework.
It has no outside dependencies, and works to have a lot of bleeding edge features like websockets, non-blocking events, IPv6 and concurrent requests.
Mojo::UserAgent
is the client that is part of Mojolicious, and it’s got all sorts of cool features:
- DOM parsing
- text selection via CSS selectors
- For example, “give me all the text that is
#introduction ul li
.” - Command line:
mojo get mojolicio.us '#introduction ul li'
- For example, “give me all the text that is
- JSON parsing
- JSON pointers
- JSON pointers look like XPath as a way of specifying data in
a JSON string
- JSON pointers look like XPath as a way of specifying data in
Mojolicious is based on “routes”, which look like:
The latter three are (apparently) ways of making flexible URL specifications that then return information to your app about the URL.
Sample app with Mojolicious::Lite:
use Mojolicious::Lite;
get '/' => sub {
my $self = shift;
$self->render( text => 'mytemplate' );
}
app->start;
__DATA__
@@ mytemplate.html.ep
Hello!
Mojolicious also has its own templating language that looks a lot like Mason, but Glen said you can use Template Toolkit as well (and presumably others, but TT was the only one I was
interested in.)
Full Mojolicious includes a dev server called Morbo and you can run your apps through the Hypnotoad “hot-code-reloading production server” if you don’t want to run under Apache/etc.
Another selling point for Mojolicious: They value making things “beautiful” and “fun”. Glen specifically said “Join our IRC channel. We will not be mean to you.”
Perl-as-a-Service shootout
Mark Allen
This was disappointing because I was hoping for recommendations to use or not use a given vendor’s offerings. I was hoping at least for “This vendor does this, and that one does that differently,” but all I came away with was “they’re pretty much the same.”
It’s a good sign that, as Mark put it, “getting PSGI-compliant apps into PaaS is generally pain free.”
His criteria were as follows:
- Ease of deployment
- Performance (ignored)
- Cost (ignored)
- How “magical” the Perl support is (first class or hacked together)
Why ignore performance and cost? I don’t know.
Big data and PDL
There were three sessions back-to-back about PDL, the Perl Data Language. It’s in the same space as Mathematica and R. I was disappointed because I was hoping for big data analysis outside of just number crunching. The analysis of galaxy luminosity was pretty and looked very easy to do, but it didn’t have any application I was interested in. I bailed after the 2nd talk.
My big takeaway from the talk was that I need to take a statistics
class.
Web security 101
Michael Peters gave a good intro talk on security, handwaving the tech details with examples of “This is how bad guys can get your info.”
Emphasis on not trusting your client data, but I was surprised and disappointed that he seemed to steer people away from Perl’s taint mode. He made vague reference to there being bugs with regexes and taint mode, but I don’t know what he’s referring to.
Taint mode is one of my favorite things about Perl 5, and there are (last I checked) no plans for implementing it in Perl 6. 🙁
One of the examples Michael used for an example of an attack with SQL injection used sleep()
to let the attacker find out information about the database based on timings. I asked him to write that up for bobby-tables.com.
On being a polyglot
Miyagawa gave a great overview of how he spends time in Perl, Python and Ruby, and what he learns from each, and what each language learns from the others.
Key point: Ruby is not the enemy. They are neighbors.
Things he likes about Ruby:
- Everything is an object
- More Perlish than Python
- Diversity matters = TIMTOTWTDI
- Meta programming built in and encouraged
- Convention of
!
and?
in method namesstr.upcase!
to upcasestr
in placestr.islower?
to functions that return values
- Ability to omit
self
- Everything is an expression.
- No need to type
;
(unlike Python) - Implicit better than explicit
- block, iterators and yield
- No semicolons, 2-space indent.
- (This last one gives me the creeps. 2-space indent!??!)
Naming differences between the three:
- Perl naming: Descriptive, boring, clones become
::Simple
- Python naming: Descriptive, confusing, everything is
py*
or*py
- Ruby naming: Fancy, creative, chaotic (Sinatra, Rails, etc)
- With frameworks, all the languages get creative: Django, nbottle,
Catalyst, Dancer, Mojolicious
When you’re going to borrow something from another language, don’t just borrow it, but copy it wholesale. Example: Perl’s WWW::Mechanize getting cloned as Ruby’s WWW::Mechanize.
Doing Things Wrong, chromatic
chromatic talked about the value of doing things “wrong” and embracing your constraints. Sometimes you can’t do The Perfect Job, and that’s OK, and sometimes comes out even better.
Example: chromatic wanted to do some parallel web fetching. He could have dug into LWP::Parallel
, but instead he went with what he knew: waitpid()
and shelling to curl
.
Screen scraping example:
- Obvious answer: HTML::TokeParser::Simple or Mojo::DOM
- Common: Regexes
- Lovely: Template::Extract
Parsing HTML with regex may be the “wrong” way to do
it, but sometimes, it’s the best solution.
Perl 6 lists
Patrick Michaud talked about all kinds of awesome stuff you can do with lists and arrays in Perl 6. After a bit I stopped trying to take notes and follow what he was saying and instead just let it wash over me so I could absorb the coolness.
I would really like Perl 6 to be easy enough to install for serious play. I need to get my feet back into the Perl 6 pool and see how I can help.
Tweakers Anonymous
John Anderson (genehack)
Quick overview of cool things that he has in his configs.
- “The F keys are not just to skip tracks in your music player.”
- Keep your configs in git. You will screw them up. This will save you.
- Make your editor
chmod +x
when you create a.pl
file since you know you will want to run it.
The coolest thing was this plugin called flymake
. Apparently it runs continuously, submitting your code to a compiler (or perl -c
) as you type. As soon as John made a typo on a line and moved to the next line, the error line was highlighted. He then demonstrated doing this with Perl::Critic
, which must be dog slow, but flymake
lets you adjust the frequency of checks.
Exceptional Exceptions
Mark Fowler, now at OmniTI. Great discussion of exceptions in Perl.
Returning false on failure sucks because you have to follow your failures all the way up the call tree. It’s tedious and error-prone because all it takes is one link in the chain to not propagate the error and you’re out of luck.
Using try/catch from Java.
There are three non-deprecated ways of doing exceptions in Perl.
eval
eval
is often confused with eval $string
which means to compile code. eval is a statement not a block so requires a semicolon after it. It works but it’s a pain.
Try::Tiny
- Simple extension to the syntax
- Uses
$_
not$@
TryCatch
- Has named exception variables
- Fully functional syntax
- Very fast and featureful
- Large dependency base
TryCatch is a little faster than Try::Tiny, but eval
is much much faster than either of them.
TryCatch has much more clever syntax, but looks (to me) to be more dangerous.
Mark recommends that whatever you use, you make exceptions out of Exception::Class
objects.
Notes and comments from OSCON 2011
September 20, 2011 Open source, Programming 1 comment Acme::Crap, Alex Martelli, Andrew Bayer, API, Bruce Momjian, Cornac, CPAN, Damian Conway, Damien Seguy, Data::Show, IO::InSitu, IO::Prompter, Jenkins, jQuery, OSCON, Perl 5, Perl 6, PGXN, Postgres, Robert Treat, Selena Deckelmann, Smart::Comments, Tom Christiansen, Unicode
Finally, two months after OSCON 2011, here’s a dump of my notes from the more tech-heavy sessions I attended. Some of it is narrative, and some of it is just barely-formatted notes. The target here is my own use of what was most interesting and useful for me at work, but I make them public here for anyone who’s interested.
This post is long and ugly, so here’s a table of contents:
Back to table of contents
API Design Anti-patterns, by Alex Martelli of Google
Abstract
Martelli’s talk was about providing public-facing web APIs, not code-level APIs. He said that public-facing websites providing must provide an API. “They’re going to scrape to get the data if you don’t,” so you might as well create an API that is less load on your site.
API design anti-patterns
Everyone wants an API. Take a look at the most common questions on StackOverflow. They’re about spidering and scraping websites, or simulating keystroke and mouse gestures. Sometimes these questions are about system testing, but most of them point to missing APIs for a site. The APIs may actually be there, or they may be undocumented.
You should be offering an API, and it should be easy. You are in the shoes of your users. You need this API just like they do. Even a simple, weak API is better than none. Follow the path of least resistance: REST and JSON.
Document your API, or at least consider examples which may be easier than text to programmers. Keep your docs and especially the code examples in them tested. Use doctest or a similar system for testing the documentation.
Back to table of contents
The Conway Channel
The Conway Channel is Damian Conway’s annual discussion of new tools that he’s created in the past year.
Regexp::Grammars is all sorts of parsing stuff for Perl 5.10 regexes, and it went entirely over my head.
IO::Prompter is an updated version of IO::Prompt which is pretty cool already. It only works Perl with 5.10+. IO::Prompt makes it easy to prompt the user for input, and the new IO::Prompter adds more options and data validation.
Data::Show is like Data::Dumper but also shows helpful debug tips like variable names and origin of the statement. It doesn’t try to serialize your output like Data::Dumper does, which is a good thing. Data::Show is now my default data debug tool.
Acme::Crap is a joke module that adds a crap function that also lets you use exclamation points to show severity of the error.
As with most of Damian’s joke modules, you’re not likely to use this in a real program, but to learn from how it works internally. In Acme::Crap’s case, the lesson is in overloading the ! operator.
Back to table of contents
Cornac, the PHP static analysis tool
Cornac is a static analysis tool for PHP by Damien Seguy. A cornac is someone who drives an elephant.
Cornac is both static audit and an application inventory:
Migrating to PHP 5.3
Gives a list of extensions. Maybe Perl::Critic should include an inventory of modules used? Elliot points out that you can give perlcritic the --statistics argument for some similar stats.
Found three different classes with the same name, but three different source files.
Summary of classes and properties makes it easy to see inconsistencies.
Has an inclusion network, like my homemade xreq tool, but graphical:
Most interesting of all was finding out about two other PHP static analysis tools: PMD (PHP Mess Detector) and PHP_Depends.
Back to table of contents
Using Jenkins
Andrew Bayer, @abayer
Slides
Andrew was clearly aiming at people who had many Jenkins instances, which we certainly won’t be at work, but he had lots of good solid details to discuss.
#1 Use plugins productively
#2 Standardize your slaves
If you’ve got more than a couple builds, you’ve probably got multiple slaves. Ad hoc slaves my be convenient, but you’re in for trouble if they have different environments.
Use Puppet or Chef to standardize what goes on the machines. Have Jenkins or your job install the tools. Or, you can use a VM to spawn your slaves.
Whatever method you choose, just make sure your slaves are consistent.
Don’t build on master. Always build on slaves.
#3 Use incremental builds if possible
If your build takes 4-8 hours, you can’t do real CI on every change.
If you’re integrating with code review or other pre-tested commit processes, you want to verify changes as fast as possible.
Incremental builds are complementary to full builds, not replacements.
#4 Integrate with other tools
#5 Break up bloat
#6 Stick with stable releases
#7 Join the community
Back to table of contents
Learning jQuery
I was only in the jQuery talk for a little bit, and I was just trying to get a high-level feel for it. Still, some of the notes made things much clearer to my reading of jQuery code.
$ is a function, the “bling” function. It is the dispatcher for everything in jQuery.
jQuery should get loaded last on your page. Prototype uses the $ function, and will eat jQuery’s $. But jQuery won’t stomp on the Prototype $ function.
Put your Javascript last on the page, because the <script> tag blocks the rendering of the web page.
Back to table of contents
MVCC in Postgres and how to minimize the downsides
Bruce Momjian, Presentations
This turned out to be 100% theory and no actual “minimize the downsides”. It was good to see illustrations of how MVCC works, but there was nothing I could use directly.
Why learn MVCC?
Core principle: Readers never block writers, and writers never block readers.
(Chart below is an attempt at reproducing his charts, which was a pointless exercise. Better to look at his presentation directly.)
Four different numbers on each table drive MVCC:
Back to table of contents
(Re)Developing Perl 5 Modules in Perl 6
Damian Conway
Perl isn’t a programming language. It’s a life support system for CPAN.
Damian ported some of his Perl 5 modules to Perl 6 as a learning exercise.
Acme::Don’t
Makes a block of code not get executed, so it gets syntax checked but not run.
Perl 6 implementation
Lessons:
IO::Insitu
Modifies files in place.
Smart::Comments
Perl 6 is solid enough now. Start thinking about porting modules.
Back to table of contents
PostgreSQL 9.1 overview
Selena Deckelmann
Slides
New replication tools
SE-Linux security label support. Extends SE stuff into the database to the column level.
Writable CTE: Common Table Expressions:
A temporary table or VIEW that exists just for a single query. There have been CTEs since 8.4, but not writable ones until now.
This query deletes old posts, and returns a summary of what was deleted by user_id.
Per-column collation orders
Extensions: Postgres-specfiic package management for contrib/, PgFoundry projects, tools. Like Oracle “packages” or CPAN modules. The PGXN is the Postgres Extension Network.
K-nearest Neighbor Indexes: Geographical nearness helper
Unlogged tables: Only living in memory, for tables where it’s OK if they disappear after a crash. Much faster, but potentially ephemeral.
Serializable snapshot isolation: No more “select for update”. No more blocking on table locks.
Foreign data wrappers
Back to table of contents
Pro PostgreSQL 9
Robert Treat, OmniTI, who are basically scalability consultants
pgfoundry.org is other stuff around postgres.
pgxn.org is for 9.1+ extensions
Use package management rather than build from source
Versions
pg_controldata
gives you all sorts of awesome detailsrecovery.conf is in the PGDATA dir for standby machines
pg_clog
,pg_log
andpg_xlog
are the main data logging files.You can delete under
pg_log
and that’s OK.Trust contrib modules more than your own code from scratch. Try
to install contrib modules into their own schemas.
Configuration
work_mem
checkpoint_segments
maintenance_work_mem
max_prepared_transactions
wal_buffers
checkpoint_completion_target
Hardware for Postgres
Don’t replace multiple spindles with a single SSD. You still want redundancy.
Backups
Logical backups
pg_dump
is your friend, andpg_dumpall
for global settingsPhysical backups
Tarball
Back to table of contents
Perl Unicode Essentials, Tom Christiansen
http://98.245.80.27/tcpc/OSCON2011/index.html
Perl has best Unicode suport of any language.
Unicode::Tussle is a bundle of Unicode tools tchrist wrote.
5.12 is minimal for using
unicode_strings
feature. 5.14 is optimal.Recommendations:
21 bits for a Unicode character.
Enable named cahracters via \N{CHARNAME}
If you have a DATA handle, you must explicitly set its encoding. If you want this to be UTF-8, then say:
Tom’s programs start this way.