CandleCRM Weekly Project Updates

2016-08-22 · ~3,514 words

2016-08-01

Hi everyone. Nothing terribly interesting happened this week, just lots of bug fixes, stability improvements, and enhancements to the logging system. As far as I can tell from logs, there haven't been any crashes that affect users in a few days now, so that's definitely good.

People new to graphs might be interested in checking out this blog post on graph algorithms, by Neo4j:

https://neo4j.com/blog/graph-search-algorithm-basics/

Anyone new to launching code in production should probably check out this list, it's been very helpful. Other resources like this would be appreciated, if people know of any.

http://yellerapp.com/posts/2015-03-03-production-clojure-checklist.html

2016-08-08

Hi everyone. The server has been pretty stable this week, and I've continued working down the list of bug fixes and UI improvements. Unfortunately, my time has been limited due to EA Global, but on the plus side, it's usually much easier to make fixes and tweak existing code than to build new systems or rewrite them from scratch. In terms of AWS resources, things have been running around $2 per active user per month, which is a good place to be - higher and it might eat into profit, and lower would mean I was too conservative in choosing machine learning models.

2016-08-15

Hi everyone. Unfortunately, I discovered a significant bug in how data was stored last week. Suppose email E1 is loaded into the database. This email has some text, which I'll call T. The graph then looks like:

E1---->T

A few seconds later, a duplicate email, E2, is also loaded. The graph is then:

E1---->T E2---->T

The reconciler passes over, sees that E1 and E2 are the same, merges them, and marks E1 as having been reconciled, like so:

E1*---->T

Key point: since an email can only have one body, the two T nodes are merged into a single T. This is different from, eg., addresses, since a single person can have more than one address.

Since E1 has now been reconciled, the NLP scanner then picks it up, and turns the raw text T into the processed, "rich" text P:

E1*---->P

After this happens, a third duplicate email is introduced, E3:

E1*---->P E3---->T

The reconciler then soon detects a match between E1 and E3. Unfortunately, the merging for these matches was tricky, and frequently had bugs. If T replaced P, then the email might accidentally be left unscanned. If P replaced T, then the email might accidentally be scanned twice.

The solution I've found is to have separate edge types for scanned text and unscanned text, so the graph then looks like:

P<----E1*---->T

The advantage of this is that the old, raw information is preserved. If there's a machine learning bugfix or a new NLP model, as there often is, new Ps can be reconstructed from existing Ts; the email and its data don't have to be wiped and rebuilt from scratch.

Unfortunately, this change requires porting over the existing database to the new format. This is somewhat annoying, and since I'd rather just do it once, none of this week's bugfixes have been uploaded to production yet; I'm waiting to see if there's anything else in the current bug list that requires moving existing data. That way, I can do all of the uploading and data-moving in one step. (Data migration is something that seems like it might keep coming up, so I've also built a few quick tools to make it easier.)

One thing that has been implemented this week, although it (hopefully) won't be visible to users, is using Tarsnap (https://www.tarsnap.com/) as a remote backup service. This has the advantage of being both a) automatically encrypted, so the database can't be read or leaked by the backup provider, and b) de-duplicated, so that any information found in both today's backup and yesterday's backup is only stored once. There are now two layers of backup: one done hourly to a different machine within AWS, and one done daily to Tarsnap's remote storage service.

I've also had a few calls this week about application ideas, ie. specific markets to target within the CRM space. This is still at the very beginning, but I expect it will gradually ramp up as I get more ready for a full launch.

2016-08-22 — ALREADY PUBLISHED (see text/newer__2016-08-22-weekly-project-update.txt)

Hi everyone. This week, I made a bunch of improvements to the UI, and to several of the machine learning systems. Unfortunately, these still haven't been uploaded to the public-facing server; the new code isn't compatible with the old ML models, and not all of the new models have been trained yet (there are a bunch of new features, so I'm making new training data sets with them included). Sorry about that. But I'll try to walk through some of what I've done, and why I think it'll be useful.

First, while there are many types of information you can extract from text, one type that's found in almost everything is dates and times. Dates come in a huge number of formats - "January 23rd, 2016" is the same as "1/23/2016", which is the same as "23-01-2016", which is the same as "this Saturday" (if, at the moment, it's the week before the 23rd). Fortunately, the date parser is pretty clever - not only does it handle all of these, but it makes inferences based on what the date and time is right now. For example, if I write "let's meet tomorrow, at 8 PM", the parser will automatically infer that (right now) tomorrow is Monday, August 22nd, so the meeting goes on the calendar at 8/22/2016, 8:00. But, the inference is so aggressive that the machine learning algorithm downstream can't tell how precise the original date was. For example, if I write "next year", I'm probably talking about a very broad range of time. However, both "next year" and "August 21st, 2017" will get parsed to the same result (since the current date is August 21st, 2016, "August 21st" is interpolated into "next year"). Of course, that creates problems for reconciliation, as well as things like weighing how important different events are. So I went into the parser's source code, and figured out the (undocumented) functions that returned which parts of each date had been extrapolated; these date flags are now some of the new machine learning features. (In turn, these flags let me create a new "Agenda" tab - listing details of future events in chronological order - without it being full of junk. In the development version, I've now set this as the default homepage.)

Second, I managed to make the candidate generation system much broader, without making it impractically inefficient. Candidate generation is the first step in reconciliation - if there's a pool of, say, 10,000 person names, I want to know which pairs of names might possibly be the same guy, so I can feed those pairs into the machine learning system. An easy way of doing candidate generation is writing queries like this:

match (n:person)-[:date-of-birth]->(d)<-[:date-of-birth]-(m:person) return n, m

which returns all pairs of people with the same date of birth. But of course, lots of dates and so on aren't exact, so this approach misses many possible pairs. What you'd like to do is walk down the list of birthdays (or names, or locations, etc.), in sorted order, keep a "sliding window" of all the elements within a certain range of each other, and flag those as possible matches. Unfortunately, I don't think this is possible inside neo4j queries. But since reconciliation is now done continuously, only a few items will ever be non-reconciled at a time. So I split the search up into a two-part neo4j query: first, find the birthday (or whatever parameter) of all people who haven't yet been reconciled; second, search the neo4j index for a range around each birthday, and flag any possible matches. This requires a separate lookup for each person, but since each index range lookup is only log(N), it's still acceptably fast.

And of course, there's been the usual assortment of bug fixes, UI tweaks, CSS improvements, and so on. (Maybe an experienced front-end dev would disagree, but I still think CSS is badly designed - half the time I'll think something like "oh, this div needs to be shifted down", so I'll duly write "div#foobar { margin-top: 20 px; }", and the browser then just ignores this and doesn't tell you why.) Just as one example of the kinds of bugs I've found, the frontend Javascript has to retrieve certain facts about each person in the CRM - their name, their email address, where they live, and so on. These results are grouped into rows, with one row for each type of fact, and the rows are numbered and sorted so they appear in a pre-defined order. In testing, this worked great. However, when I added some more rows, it broke. The reason turned out to be that the row numbers were returned from the database as strings, not as integers. If you have this set of strings, one string for each row:

"4" "2" "8" "1" "3" "7" "5" "6"

sorting the rows returns:

"1" "2" "3" "4" "5" "6" "7" "8"

which is perfectly fine. But if there are eleven rows:

"9" "4" "2" "11" "8" "1" "3" "10" "7" "5" 6"

then string (alphabetic) sort stops being the same thing as integer sort:

"1" "10" "11" "2" "3" "4" "5" "6" "7" "8" "9"

and then everything breaks. Fortunately, it's easy to fix once you know what's going on.

2016-08-29

Hi everyone. I'm continuing to tweak the machine learning algorithms - right now they're too "lumpy", and merge many unrelated people together. I could just lower the score threshold for merging, and I might be forced to do that next week. But I'm going to try and see if I can solve it by making the models more generally powerful, without introducing lots of false negatives. In the meantime, I've uploaded all the new code from the last three weeks to the public server at candlecrm.com, since there's also a bunch of UI improvements and general bug fixes and stuff, and I didn't want to delay those any further.

2016-09-05

Hi everyone. Good news: the excessive lumping problem is now (mostly) solved. I've added another layer to the machine learning models, and I've also made some tweaks to my training-data-collection program, making it faster for me to classify each pair as "match" or "not match". This lets me create larger training data sets for each model, in less time.

I've also re-written the edit menu's code to create a simple way of editing a node's links, in addition to editing a node's properties. For example, a Person node might look like this:

Name: Bob Smith Email: bobsmith@yale.edu Phone number: (203) 123-4567

Each of these facts is just a string that's a property of the Bob node, so editing them is (relatively) simple. But we might also want to store some other facts about Bob:

Employer: Yale University Lives in: New Haven, CT Attended: Burning Man 2015

These facts aren't properties, but links to other entities (eg. "Yale University"), which themselves have their own properties like names, locations, websites, and so on. Suppose Bob moves to Bridgeport, CT, and the user now types in:

Lives in: Bridgeport, CT

We certainly don't want to rename the whole city of "New Haven, CT" to "Bridgeport, CT". Instead, what the edit menu now does is a) recognize that the link between "Bob" and "New Haven" is obsolete (since the user edited the original text); b) recognize that "Bridgeport, CT" is a city; c) create a Lives-In link between Bob and a newly minted entity of "Location: Bridgeport, CT"; d) reconcile the new "Bridgeport" with any existing "Bridgeports" in the database. This does require that each linked entity have a unique identifier - eg., it wouldn't work if Bob named his dog "Fido", since there are probably many other "Fidos" - but I think it's pretty good for most purposes.

There's also the usual assortment of bug fixes, UI tweaks, CSS improvements, etc. There are dozens of these, so I can't really list them all out, but one I especially like is the arrow icons in the previous/next buttons. These used to be done with ASCII art, like so:

| <-- Previous | Next --> |

These buttons now use real arrow graphics from Font Awesome (http://fontawesome.io/), who have generously made all their icons available for commercial use.

2016-09-12

Hi everyone. Good news - as you might have guessed, I'm not a web design expert, and I thought I'd have to hire someone to really make the app look "professional". But this week, I switched to using Bootstrap (getbootstrap.com) as my CSS framework, and I think this solved many of my problems. It's not a magic bullet, but I think it really makes it easier to at least make an app presentable, without having lots of design experience. (Though, of course, that would still be nice!)

I've also added a "notes" box, where people can write in their own information about their contacts, since that's a pretty critical feature. The notes are read into the graph like emails are, so it's also a faster way to enter information. Here are some screenshots:

http://imgur.com/a/frBLQ http://imgur.com/a/QU1C1

(you might notice that the new event has no title - that's unfortunately a Javascript bug. Sorry, shouldn't be too hard to fix though)

I've also tweaked the sentence-splitting algorithm to be better adapted to email text. Many emails (especially automated ones) don't really have punctuation, they just have big lists like:

Event #1 Name Event #1 Date Event #1 URL

Event #2 Name Event #2 Date (....)

and by default the whole list is read as one giant sentence, which screws up several algorithms. This is (mostly) better now, though.

More strategically, several people have told me that they think Salesforce is essentially a monopoly in CRM, and I'd have to integrate with them to have any chance in the market. I'd like to integrate with Salesforce, of course, but I don't think it's necessary for an MVP. The market share data I found says that Salesforce is just 20% of the total, and 55% is owned by "Other" (ie., not the top five vendors):

http://blogs-images.forbes.com/louiscolumbus/files/2016/05/CRM-Market-Share-2015.jpg http://www.crmsearch.com/images/crm-market-share.gif

This data is also weighted by revenue; Salesforce's marketing is mostly towards large enterprises, and so I'd suspect it'd be even smaller if weighted by user count. I could be wrong, of course. But I'd prefer not to depend on another company that owns the customer, like the position Zynga is in with Facebook.

2016-09-19

Hi everyone. I'm currently getting ready to announce CandleCRM for beta testing. I'll put out a public announcement on Facebook and so on, so that anyone who's interested can try it out. I've added an invite-code field to the signup form, so that people can try it out a few at a time, because:

I wanted to announce it tonight, but unfortunately, it turns out that one type of database error still requires a manual fix, which obviously won't work if there are dozens or hundreds of people trying the software. Oh well, shouldn't be too hard to solve. A bit more time will let me add more info to the public-facing site, which I should've done before anyway.

2016-09-26

Hi everyone. I've announced CandleCRM on my Facebook for beta testing, and have made a spreadsheet listing each batch of new people to add. Since I've already "invited" everyone CCed here, the signup info is:

https://www.candlecrm.com/YZm6sB.html Invite code: PRETZEL

Bugs caught so far are mostly random edge cases, like not failing gracefully when trying to sign up for an account under one email address while being logged into GMail under a different address.

I'll also be applying to Y Combinator for the coming winter batch (Jan. - Mar. 2017). Wish me luck :)

2016-10-03

Hi everyone. I've now invited all the people who signed up for the CandleCRM beta test. The Graylog daemon alerts me whenever there's a server error, so I've been finding and fixing problems as they've come up. I think the "critical" errors have been mostly fixed now, which is encouraging.

Since people can now try using CandleCRM, I think it's time to shift gears somewhat. Whenever I try a new piece of software, around half the time I'll wind up muttering to myself about how the authors obviously never watched anyone try to use it, or else they'd have done X. Since there's a million different things I could improve next - which is a big part of the excitement, there's just so much opportunity - I will now follow my own advice, and do in-person testing so I can figure out which parts to focus on.

If you're in the Bay Area. and have an hour or so free this week, I'd definitely appreciate letting me sit next to you while you tried CandleCRM, since that's the fastest way to figure out what's broken or confusing or missing. (If you don't have any strong reason to use a CRM, you can of course just delete the account afterwards.) Thanks!

2016-10-10

Hi everyone. I'm gradually working my way down the list of feedback I've received; since it's generally easier to fix non-ML issues than ML issues, I've focused most of my effort there. Eg., multiple people said that it was annoying to not be able to use the Back button to navigate between different tabs. That should be fixed now for every major browser except IE8, which sadly doesn't support HTML5. On the ML side, the next step is probably adding categorization - labeling companies as big or small, labeling emails as personal or commercial, and so on.

Right now, I'm working on expanding the authorization code, to use other email providers besides GMail. Fortunately, many different companies use a standard called OAuth2, which lets me ask for app permissions without collecting peoples' email passwords (obviously, a potential security risk). Unfortunately, when I followed Google's instructions for using OAuth2 to connect to GMail, they directed me to the Google OAuth2 library, which (surprise surprise) turns out to have lots of Google-specific code. Oops. There are other open-source libraries I can use here, so this shouldn't be too hard to fix.

2016-10-17

Hi everyone. This week, in addition to the usual bug fixes, there were several standardizations I made within the code base:

Next week, I plan to focus much more on machine learning, as beta testers have widely agreed that accuracy is too low. I have several ideas for how I can increase model performance; the most obvious is using gazettes, which is basically just a pre-made list of words that you know are likely to fall into some category. Eg., everyone knows that a phrase ending in "Inc." or "LLC" is likely to be a corporation. Stanford NLP talks about how to use these, but doesn't use them themselves, which is pretty strange; maybe the academic performance contests they enter forbid them? Oh well, more performance for me.

2016-10-24

Hi everyone. I've added a "tag" system, so that objects in the database (people, places, events... ) can be labeled with categories (Friend, Work, New Lead, Important, etc.). The next step is to auto-infer a set of default tags with machine learning; this should really help cut down the number of "events" that aren't meaningfully events, since those can be auto-tagged as "Junk" or "Random" and made invisible by default. The next step after that, of course, is to use the tags that each person makes themselves as training data, so they can tag some things and then have the system infer the rest. I think I'll hold off on that, though, until there's a larger and consistent user base, since that's only useful once people are willing to spend a bunch of time tagging stuff.

2016-10-31

Hi everyone. I got invited to Y Combinator interviews; since I've been busy preparing for that, I unfortunately haven't done any coding this week. Interviews are this Thursday, so on the downside there's not a lot of time to prepare, but on the plus side at least it's a quick decision.