thoughts & updates| jpt.sh
(Notes from a brief introduction I gave at the 10th Annual Civic Scopeathon, put on by my students. Someone asked me to share it, so putting it up here without any real editing.) Civic Tech is a young field, about 20 years old. That means Scopeathon has been around for about half of that time. I want to take a second to talk about what has changed in civic tech in those past twenty years, and why Scopeathon embodies some of the most important ideals.| Posts on jpt.sh
A while back I got asked to do a usesthis.com interview, and it is now up here: https://usesthis.com/interviews/james.turk/. I’m a big fan of usesthis.com and other sites that focus on tools & this was fun to do. A lot has changed between when I wrote this & now, I suppose that means I should write a new post.| Posts on jpt.sh
I’d had this draft ready and fortuitously found out today is World Wide Web Day even though nobody knows why. We tend take the web for granted, so it is nice that there is at least a nominal day for it. (The web is only 35 and it took Earth a few billion years to get a day named for it.) The web is an incredible achievement, an attempt to build a “universal linked information system” that has surpassed the ambitions of its creator. But it is also under threat, with a few big players con...| Posts on jpt.sh
Back in 2010, my colleague Michael created a Python library with C implementations of a bunch of common string distance algorithms. I remember wanting to call it strfry at one point but jellyfish won in the end. A year or so later I eventually took over maintenance of the library and didn’t give it much thought for a while & certainly didn’t expect it to become the most widely-used library I currently maintain.| Posts on jpt.sh
Media Party 2023 Presentation: Web Scraping with GPT Media Party is a conference that “facilitates the intersection between the media and technology industry in an innovative, international and hands-on way.” This year was their first one in the US, and it was conveniently here in my backyard. The organizers were nice enough to invite me to speak on the work I’ve been doing recently on web scraping with GPT. If you’re interested, my slides are available here: Automating Web Scraping w...| Posts on jpt.sh
A couple of days ago I wrote about using GPT-4 to automate web scraping. I was surprised at how well it worked for a first attempt and have been curious how far I go to make it useful. I’m building an experimental interface here if you’d like to follow along: https://github.com/jamesturk/scrapeghost Here’s what I’ve been working on: Command Line Interface If you’d like to just quickly experiment with the tool, I’ve added a command line interface.| Posts on jpt.sh
(Part 2 now available: Automated Scraping with GPT-4, Part 2) Like most people I know, I’ve been watching the pace of improvements to LLMs like ChatGPT and GPT-4 with a mix of awe and trepidation. I’ve been wanting a small project to get to explore the APIs better, and recently decided I’d try to see if I could use it to automate web scraping. For context, I’ve written a lot of web scrapers. For the better part of thirteen years, I ran Open States, a project that scraped state legisla...| Posts on jpt.sh
I’ve been a vim user for more than 15 years. Before then I would try a new editor every few months, Visual Studio, Komodo IDE, Kate, and JEdit are a few that I remember sticking with for a while before hitting some wall and looking elsewhere. So when I eventually took the time to learn vim and found that I really enjoyed working in it, I figured I was done switching editors.| Posts on jpt.sh
What We Lost If you’re around my age, you remember an internet dominated by niche message boards. Gaby Del Valle’s article did a great job capturing what was special about them & how they differ from present day social media. In my teenage years, I was very active on the old GameDev.net. I wound up becoming a moderator, spending my time answering questions, talking about nonsense, and very very occasionally publishing a small game. I stuck around for a while even after I mostly lost inter...| Posts on jpt.sh
Earlier this week, Django 3.1 beta 1 was released. While the big news is the addition of asynchronous views and middleware, there’s a tiny feature I managed to get added that I wanted to bring a bit of attention to as it makes writing apps that use Postgres’ full text search a little easier & nicer. It’s a small line in the changelog, but the SearchQuery constructor now accepts the argument search_type="websearch". This allows you to write code like:| Posts on jpt.sh
May 29th, 2007 On May 29th 2007 I’d been living in DC for a little under a week. I liked it quite a bit, which was lucky as I’d made plans to move there sight unseen. In December of 2006 I’d accepted a position with Sunlight Foundation, an up-and-coming nonprofit with a mission I cared about and which already seemed to place an importance on technology. In fact my position was listed as being within Sunlight Labs, “a Sunlight foundation pilot project to prototype tech ideas”.| Posts on jpt.sh
Like many developers, I’ve written my own blog software half a dozen times. I’ve probably spent a lot more time tweaking the software than writing posts so this time I decided to do something different. I wanted to keep things as simple as possible but still have an excuse to learn a few new things. I wound up settling on Hugo hosted on Amazon S3 w/ CloudFront in front of it. One of the deciding factors that made CloudFront come out on top is that I wanted to see what it’d take to get t...| Posts on jpt.sh
Note: Another old one that I thought was interesting enough to migrate. I’d meant to keep doing this every year. Maybe next February? Earlier today I noticed an email I was looking up was from exactly three years ago. From there I wound up on a random path that had me reading emails from February 24th for the past 6 years. It wound up being a pretty interesting look at what I’ve been doing over the last few years that I thought I’d write up for my own benefit.| Posts on jpt.sh
Note: An old post I brought over because at least two people started small dotfiles projects based on this :) I now use chezmoi for this! If you find yourself logging into 3 or more machines in a given day (for me my laptop, work machine, and a few servers) you’ve probably faced the problem of being on a machine and not having some essential piece of functionality in your .vimrc or .bashrc alias handy. Maybe you copy the file you need over with scp, but this is never satisfying as it will g...| Posts on jpt.sh
Note: an old post from 2008… not the best post but a nostalgic look back at my introduction to organizing developers. This past weekend I had the pleasure of attending the (fantastically organized) Rochester BarCamp 3 on the RIT Campus. A BarCamp is basically a group of knowledgeable people that get together and give talks on subjects that they care about. It's quite possibly one of the coolest things I've had a chance to be involved in the entire time I've been at RIT. (I didn't hear about...| Posts on jpt.sh
I’ve spent nearly twenty years working in civic tech. It feels like with everything going on right now, I should have something to say… This is my attempt to put into words what’s been swirling around in my head during this chaos. This is my perspective, which I know is just a small part of the picture. In particular, I am focused on the US here, and while I’ve worked with many civic tech nonprofits, journalists, academics, and at commercial civic tech firms, I should note I’ve neve...| jpt.sh