Introducing a new show from Rocketship.fm!

This New Economy - A Podcast Exploring Humanity Through Economics

Listen Now

Release Date 10/13/15 Episode Length 31:41

Jason Amunwa of Filament

Iterating Your Way to Product-Market-Fit

with Jason Amunwa of Filament

Rocketship: Jason welcome to the show.

Jason: Thanks you all, good to see you guys, thanks for having me here.

Rocketship: Tell us a little bit about what you’re working on at DT and with Filament.

Jason: We’re working on Filament, it’s a blog analytics tool or rather an analytics tool designed specifically for blogs. It came about out of our own frustration just using traditional web analytics tools to try and find answers to what we thought were simple questions but actually difficult to get to and pull out easy insights that we can act on.

Rocketship: I know, you can go into Google analytics looking for answers and specific to blogging. I know it’s really hard to find what you’re looking for. Especially when it bridges across different social platforms and multiple posts and try to really get a good feeling of what’s working in your content strategy.

Jason: Totally, I mean it’s, we do UX design and Google analytics is like our bread and butter. We constantly found ourselves asking why. It’s like, why did we a traffic spike. Why did that post get shared a lot and that other post didn’t get shared. We just went on a bit of a vision quest to try and figure out how we could answer those questions quickly and definitively. Because they’re pretty interesting questions. I want to know why a post got shared more than the rest of my posts. We’re finding that the tools just present the data, but just very upfront with no interpretation. That’s where we’re trying to fill the gap with Filament is help people get to why behind what’s changing on their site quicker.

Rocketship: How has it changed? You guys have been building it for you know, going on a year now.

Jason: Probably a year or so.

Rocketship: Since you’ve been able to use it, how has it changed the way that you guys are tracking and working on your content?

Jason: I think mainly it’s helped us with focus, honestly. Trying to figure out, okay which pieces of content are doing really well and why are they doing well. Things like where should we promote them, who are the people that are promoting that content and trying to connect with them. Also where they’re posted and they’re under-performing and could use a little love. Maybe people abandoning it halfway through. You need to optimize the imagery or something like that. It’s really been just helping us find opportunities to grow with our own blog. That’s where it all started, is we wanted to grow our blog. We knew that content marketing is ascendant right now. We needed a tool to help us do it quicker.

Rocketship: You guys have been building this for a year and I know you has some private beta users in there before you opened it up to the public. Walk us through that process and what you were looking for with those beta users before you were ready to open it up.

Jason: It all started with spreadsheets honestly. We had an idea that when you’re looking at the analytics for your blog, there’s many number of metrics that you can look at to try and figure out like user engagement or visitor engagement. Traffic is one of them, but a page doesn’t really tell you very much by itself. It’s just a number honestly. When you start looking at different metrics in combination, you can start to find interesting insights. Things like the number of shares a page gets. Again that’s not just a number by itself, if you combine that as a ratio with page views, you get a share ratio which lets you compare pieces of content to each other in terms of what their share-ability is. We just started messing around in spreadsheets with a lot of the stuff to try and see what is useful to us. It can be like trying to drive a car with ten different dials on the dashboard and someone asked you, “How’s the trip going?” You could point to any different, one of those dials and the answer is a bit muddy and blurry. We started experimenting with combining all of those metrics into just one score. We call it an engagement score. People can create their own combo scores based on what they think is important. What we found is that by tracking that single number over time, we’re able to start getting a much better sense for how the blog was doing overall. Because if you baseline that score and then you’re able to see when there’s a significant movement in it, it tells you overall what’s happening with your blog. If the trend is up into the right, then you’re doing something right. We had all this different links, spreadsheets running together just running the math and we were updating them by hand everyday. Once we started seeing this algorithm, this formula give us useful insights, we just started building an automated system around it. Really we started super small, we dogfooded it and started using that system on our own blog. Then we invited a handful of other folks that we thought might find it interesting in to try out the prototype. I think we built the prototype in about a month back in December last year. We quickly ran into scaling issues. Because the amount of data that we’re capturing just overwhelmed our little prototype. That’s when we realized, when we started looking at it as a product and as a business we’re like, “Wow, this could be something that really fits in with our vision of helping everybody make the web better.” Because if you know how to make your blog better, that could impact all blogs and better blogs for everybody will make a better experience for everybody on the web. I’m talking a lot, are you guys still there?

Rocketship: Yeah.

Jason: You’re very good listeners. We started just letting a small number of people in. I think our first group of beta users was like 10 people. We spent a couple of weeks just going out and trying to recruit those people by hand. Just explaining the vision honestly. Explaining that it’s a prototype right now, but here’s where we see it going. We just gradually built up our beta group from there. We had an existing audience based on other products that we’ve done in the past. We started cherry picking the folks that we felt got the vision and could help us guide the product towards it with their feedback. That kind of kept out, we had about maybe a couple of hundred beta users before we decided to build a beta V2 that could handle the load. Because again we started hitting scaling issues. I think our first scaling issue was just my sanity, because I couldn’t update spreadsheets fast enough to keep up with all of the data that we’re collecting. Then our second scaling wall was with our first prototype when you know, just collecting so much data and it was just overloading our database.

Rocketship: Just to go back to something you said with you know, you’re handpicking people that you thought align with your vision. What did you do when you saw people either using it in a different way than you intended or not connecting with it in the way that you had hoped?

Jason: I always ask them why. Anytime it’s different, and even when things were going in line with our expectations, you ask why. It’s just having that curiosity and a desire to know. Anytime we saw somebody using it a lot, I mean we strung up our prototype with analytics like mix panel and intercom to try and track how people were using it. We just asked the question why a lot. Why are you visiting this so much or why haven’t you been visiting it. Is there something that’s missing, was it difficult to set up. Just trying to get to the why in any scenario honestly is what really helped us. Because the more we understand about the way people are reacting to it, the more in terms we are with the decisions we’re making.

Rocketship: Can you think of an example of one of your beta users that you had to reach out to, to ask why and how that changed the product?

Jason: Yeah. There was one user that we connected with who began using it quite a lot. When we talked with him, he was saying, “I’m using it tons, it’s great.” When we looked at his user engagement stats we saw that there was one log in over significant period of time, I think over a month. We have this real disconnect. We were like, “He says that he loves it and uses it all the time.” Then we see his stats and like, “Yeah, he’s visiting a lot of screens but we don’t see him logging in more than once a month or so.” Just pinged him a note and it turns out that he loved it so much that he actually bookmarks the tab and just has the tab, has fill it open in a tab all the time and he’s constantly logged in. We had set out an expectation that we would classify people who logged in like three times or more in the last two weeks is engaged users. His behavior in using the part was completely different that he was still a really engaged user. Had we not talked to him we would never found that out or we would’ve thought he’s just pulling our leg. That was a pretty interesting finding.

Rocketship: When you’re trying to find private market fit, how long did that take you guys after you put out the first beta to those first few customers and have you found it yet.

Jason: I was going to say, I’ll tell you when we find it. We’re iterating towards it and we’re trying to be as methodical as possible in moving towards it. In terms of how long. I mean how long is a piece of string. It takes as long as it takes. It’s funny, we talk about PNF like it’s the solid milestone. It’s quite a nebulous thing. There are a bunch of different definitions for it. I don’t know my puzzle feeling up here, I know it’s kind of like corn. You know it when you see it, but it’s very difficult to concretely define it. We’re still chasing after it. We’re trying to be as methodical as possible. We just set up a trial funnel, free trial funnel and we’re running that right now. Really just push people over the top and watch what they do as they go through the process and then iterate from there. Just make sure that we’re measuring that contained experience of a 14 day free trial and see how people react to it. Just asking that question, why. Why are people tending to upgrade right at the beginning of the trial or right at the end, but not in the middle. Why are some sources of traffic delivering users that are highly engaged and others are not. It’s constantly just seeing those opportunities and following it to understand why.

Rocketship: Is it difficult when you have a product, I’m assuming there’s some kind of tracking script or code that people have to install. Is it difficult when you have that as a starting point. Do you find that a lot of customers never get pass that point?

Jason: Yes. We’ve done a lot of work on our activation flow. I think this is a problem of web software in general. Anytime it has something to do with your own site, you pretty much have to drop a piece of code. That’s a necessary step for a large number of products on the web. That’s actually something that we try to tackle with Filament and the previous iteration was we made a drag and drop process. You could install apps on your site with drag and drop after you connect it to the platform. We’ve done pretty well with that activation flow. With analytics, it’s been a challenge definitely. Because we integrate with Google analytics, so we allow you to connect to Google analytics account. Then we pull your data from there and then combine it with visitor engagement data and try to present it to you in a way that is more digestible. Because GA is a huge repository of information. There’s useful nuggets in there, that often times it gets lost because you don’t know where to find it. The codes issue is something I think that’s inherent to building stuff on the web. It’s always been a difficult problem. That’s where our UX experience has come in really handy. Because we know the pain of trying to get these codes that’s on your site and what to expect or the questions that you have when you’ve set up a new analytics tool, there are a lot of questions to address.

Rocketship: You made a great comparison earlier to the dashboard of a car. You could be driving a car and see that it’s going 80 miles an hour, which is great. If you don’t pay attention to the fact that the engine is over heating, you’re in trouble.

Jason: Exactly. There’s so many different metrics to watch. It becomes tricky to answer some of the questions. Like your boss will walk into the front door in the morning and say, “How’s the website doing?” You’re like, “Ah, traffic is up and shares are down and the bounce rates level are, I don’t know how to answer that question.” We’re trying to hit that simpler.

Rocketship: When you’re building an analytics product, how do you take people who maybe that was their background of having such a treasure trove of data and not really knowing what to focus on. How do you present them with all this new data and make sure that they’re using it in the right way, in a way where they’re actually going to change the way that they’re writing content to benefit from it.

Jason: That’s a great question and that’s one that we’re still trying to answer. Mainly I think there are two approaches that we took on designing Filament. One was, we like to solve our own problems. We’ve done it with our previous products. We worked on slide net, we worked on hello bar. Both of those were products that grew out on our own need. We needed to be able to create a content slide as quickly and we needed to be able to quickly redirect traffic to a specific page. Those products were focused on solving those problems. We found out that those problems are actually valuable to other people as well or the solutions to those problems. When it came to analytics, we leaned heavily on our own experience. We looked back at basically Chuck sat me down and this is like, “What questions do I ask you that make you roll your eyes and give you a headache?” He started there honestly. Things like why was there a traffic spike on that day. How is traffic from a particular source doing today. Which posts get shared the most and which ones are the most share-able. Questions like that that typically each take a couple of hours and multiple log ins to multiple analytics tools. We looked at those questions and try to figure out, where would I go. What would my playbook tell me to do from my own experience. To answer that question and then let’s see how we could pull that information together into one place. I think the second thing or philosophy and approach that we use was, data by itself is useless. It’s like stating a fact. You had 3,000 page views yesterday. That’s good to know but unless I have a bunch of context and background knowledge about whether or not 3,000 page views is high or low or normal for my site, the data by itself isn’t useful. We try to always present the data. Even though it’s the same data that’s on Google analytics, we try to present it in context. Not just you have 3,000 page views yesterday, which is 28% higher than the previous period. The majority of that traffic came from Twitter and it was driven by a Twit by this person. We try to connect those dots together within the interface, so you don’t have to do all the mental grunt work of figuring that out yourself. That’s what really takes the time. We called it TTIs, it’s time to insight when we’re trying to close that gap. That ultimately whenever we get to our perfect vision of the product, it will be you know, you look at analytics for 30 seconds and you understand exactly what’s going on and it helps structure your day as to what you’re going to focus on.

Rocketship: I think the back and forth that you described with Chuck is something where that’s usually the perfect place to find a product idea, is in the moments when you’re so conditioned to just say, “We can’t do that or we can’t find that information.” Those are the things where you need to remind yourself to stop and try to see if there really is a way to do it. With some of the stuff, there just wasn’t a way before. I know Twitter analytics tell you one thing and it’s not very valuable and buffer analytics tell you another. Google tells you something else entirely. It’s such a tangled web.

Jason: That’s precisely why the question why is always so fought with uncertainty when it comes to web analytics. Because can you say for certain that someone bought because this other person Twitted and then you clicked the link. It’s very difficult to definitively link those events together. I don’t think that we’re there just yet. I mean this is our first take. Like I said it’s that time to insight is the problem, we’re trying to close the gap. If we keep firing on it we can close it completely and then provide that visibility. Because right now it’s very half-hazard. Even experienced content marketers will have multiple tools set up and opening in their screens at the same time. They’re picking numbers from both and then putting them together to try and find insights and then report them to their team. I was thinking about it the other day, it feels like it’s 2015 and we’re still panning for gold. We’re kneeling beside adobe screen and it’s like shaking the dirt out to find those nuggets of gold. Just seems like it’s not the right way. Especially for things that are as valuable as your website that has a direct impact on your bottom line, so why not make it easier.

Rocketship: For who’s listening, if you want to see a deep dive on the other side of this, on the spreadsheet side of it, we talked with Manuel from code-ship a while back. He shared his spreadsheet and it’s a very complex process. There’s insights that you can pull out of that, but it’s very hard to measure. Working towards Fit, something that you guys don’t feel like you’ve fully hit yet. What are you going to do in the coming months to help you get it.

Jason: It’s funny actually, I mean we were trying to figure it out. Then we realized, the people who … The source of knowing that we got Fit would be our users. We ended up publicizing our product to work map and encouraging people to vote on the features that we’re proposing. That gives us a signal. It’s not like we’re 100% tied to that roadmap, but it’s really interesting to see what people gravitate to and what gets people really pumped. What features people are like, “Eh.” That’s nice to have, but I’m not really interested. It’s really just asking users, because ultimately they’re one half of that equation. They’re the market side of it. You’ve got the product side of it, so why not ask the other side what they’re looking for. That sounds like super basic advice, but it’s helped us a lot honestly. We have an entire section where our roadmap is publicized and people can vote on it. We just can see what’s coming in and we’re watching it with great interest.

Rocketship: How did you share that with them?

Jason: We do it actually in the app. Because we want people to be thinking about what it could be, while they’re actually in it and using it. There’s like a little slide out from the left hand side where you can see a listing of all the features that we’re considering. Then it links out to just a page. We do our support with uservoice.com. It’s been a super helpful tool for us to manage that kind of stuff.

Rocketship: When we were talking earlier you mentioned that one of the big things on your list next is working on the onboarding flow. Do you find that it’s more onboarding and how to use the application or more education on what to do with the analytics?

Jason: I think that depends on the user’s background. I guess taking the 30,000 from view, we looked at web analytics in general, just the market for it and the tools that are available. We found that, what we felt that the analytics market is very polarized. Because on one hand you got Google analytics which is free in tracks, virtually everything under the sun, so that everybody has that on their site. They’re not able to use it as effectively as maybe they could if they had a lot of analytics experience. Then on the other end of the scale you’ve got super premium quite expensive enterprise level solutions. You’ve got like Chartbeat or Adobe I think swallowed up Omniture a while ago. These tools started like 5 or 600 bucks a month, growing up to 3 of 4,000. Thing like, I think that Parsley is another one. We saw this big gap in the middle. It’s like there’s gold to be had in analyzing or interpreting the analytics that would really help a growing blog, but they’re really struggling to find insights with a free tool, Google analytics. Or they have to pony up some serious monthly cash to get something a bit more custom designed and tailored to their needs. We saw that gap as an opportunity to step in and try to help people who were not ready to pay that much money but desperately want more insight and want to be able to find opportunities to grow without having to hire a full time analyst. That was the context within which we approached it. In terms of whether on-boarding helps or how on-boarding helps versus just understanding how to use the products, I mean it’s a question of marketing versus on-boarding honestly. I mean the marketing position of products and tell people is this right for you. Then the on-boarding is, you decided it’s right for you, here’s how you use it. We’re trying to use the on-boarding as a way to almost like lead people by the nose directly to the most valuable pieces of the product right off the bat. Analytics is very individualized. Everybody pays attention to different metrics, it’s their key indicator of success. It can be a bit of a moving goal or moving target to hit. What we’re trying to do is show people, this is where you can get a lot of value out of Filament right off the bat. Then we evolve it over time. We have a 14 day trial so over time, it starts to reveal more and more points of interest inside the product. We just try and present it to the user in as friendly way as possible and show them, we’ve tracked this on your site, this could be interesting to you.

Rocketship: Interesting. You’ve opened up the product to the public about a month or 6 weeks ago. Even though you’ve been working on it for a year, it’s like a new start when that happens.

Jason: It’s like all of the assumptions that you’ve held over the past year, they come into question.

Rocketship: What are you most excited about in the coming months?

Jason: I think really starting to reach back out to people who’ve been getting a lot of value out of it and finding out how they did it. Then communicating that to other people. I think with analytics it takes a while for it to start paying off. We can all remember when we first hooked up Google analytics and then you excitedly checked it the next day and you see that single blue dot in the chart. You’re like, “Yay, the data is coming through.” Then you sit back and you’re like, “Well, what do I do with this?” It’s only once you start getting week’s worth of data or a month’s worth of data that you can start to see trends and patterns come through. I think starting to hear some of the success stories that other people are getting, would be really validating for me. We’ve seen some success with using the system ourselves and that’s why we built products around it to try and help other people find the same path. I’m excited about that. I think the other thing I’m really interested in doing is, I can’t wait to just hire our data scientist honestly. Because we’re collecting a lot of data and I want them to mine through it and find super useful things that we can communicate to the users. Because that’s when you start having an impact on everybody. Because we’re finding best practices based on the trends and patterns that we see.

Rocketship: Very cool. Jason, tell everyone where they can keep up with you and with Filament online.

Jason: You can find Filament at filament.io and they can keep up with me, best place is probably Twitter honestly. My handle is king_jaffy, J-A-F-F-Y.

Rocketship: Awesome, thanks so much for coming on.

Jason: All right, thanks Tess, it’s been good.

Rocketship: That’s it.

Jason: Cool. How’s the …

Transcription provided by Jotengine

SHARE