(survey result) Almost all

Sharing information

Almost all designers use Standups (survey result)

Organization Almost all designers use Retros (survey result) Decision making tools

Almost all designers use 2x2 (survey result) Most designers use Dot Voting (survey result)

Most designers use Stack Ranking (survey result) Most designers use Parking Lots (survey result) Many designers use Roman Voting (survey result)

Organization -> Balanced Teams

“Have a balanced team throughout the project. Engineers are important for , just like designers are important for implementation.” (survey response) Designers would like to finish with the team (Often they are rotated to start a new project before the previous project is done.) “If you were a designer being dropped on a project and you weren’t on the DNF, you’d be missing a lot.” – Participant 32 “I really enjoy involving the whole team, and you see how team members, (that maybe had no idea how to do this) get really interested interviewing the people that might use their product and they take care of every single detail during synthesis. Is very gratifying.” – Participant 52 All observed teams had at least one PM, one designer, and two engineers. Several teams had two PMs,

Roles

two designers, and six engineers. (participant observation.) Need to understand the other roles on the team PM Role Aligning stories around the crux of the product. Balancing user needs with implementation risks Ideally have a carefree attitude Comfortable working with imperfect information Client PMs out of their element with design thinking Designer Role “We (the designers) are the empathizer in chief in a balance team. It’s our role to speak for the customer strongly.” – Participant 25 Hard for engineers to understand what design is doing during a D&F “With design, there are so many facets of skillsets.” – Participant 25 Some designers do not want to be full stack: “A client designer was lamenting of like why are they making me do more work and then they’re just paying me maybe a little bit more.” – Participant 25

“[The designer] is speaking on behalf of user.”​ – Participant 25 designers need many skills to be successful: “When I look at the playbook, we are asking client designers to do so much. Other people would have a different mind-set. They’re like no, we’re asking them to do what they should be doing like tell that to the guy who’s been a designer for 10 years.” .” – Participant 25 Respecting what client designers bring to the table: “I want to be respectful of the years they’ve put into the work that they do.” – Participant 25

“There are two types of design research and problem solving. The vast majority of designers use the “genius designer” or “hero designer” where an experienced designer creates a product based on expert judgment. At , we use insights-based design. We acknowledge our assumptions and make them explicit.” – Participant 43 Appreciating age difference between pivot designers and client designers: “Many pivots are in their mid-20s and then you take someone from these enterprise companies and they’re in their late 30s, early 40s like it’s quite a position to put them in and I often remind myself that I have things I can learn from them as well and I want to be respectful of the years they’ve put into the work that they do.” – Participant 25 Learning design thinking is hard, recommends guiding student through the first time through and then slowly letting go D&F is an involved process. Integrating design thinking into workflow: “Whenever I hear about design thinking it’s like a daylong workshop (not the involved process that we follow.)” – Participant 25 Integrating design thinking into workflow: “(In comparison to other approaches) we come in, we coach, we impart and then we leave and hopefully revisit.” – Participant 25 Dev Role Involving developers in interviews and affinity mapps to help with synthesis: “We got our developers involved in that process because it’s good for them and so we run much more quickly because now all the whole team was like involved.” – Participant 25 Aligning the team

Aligning team to common goal: “Making sure that we are all aligned and working towards the same goal.” – Participant 46 Using Dump and Sort and 2x2 to align team

“Ego-less” work

Organization -> Dual Track Agile

Taking the “me” out of the work: “When you start talking about someone else, it takes the “me” out of your work and then you just have such better conversations.”​ – Participant 25 Verifying that we’re working on the right features “We are iterating and continuously learning” – Participant 44

Track 1

Characteristics

Open-ended and messy Methodical and systematic “ essentially is the practice of design thinking. And we are better off by incorporating design thinking throughout the project.” (survey response) “ is crucial to understanding a client's problem, without it we're not better off then they are prior to coming to us. We have some pivots on loan from Office B who said they were really impressed that we do on every engagement in the Office A and that they were going to bring that back an encourage the Office B Associate Director and office to start doing again.” (survey response) “It is like a snowball. In the beginning, it’s so hard to push and then we get more information and then we get more information and then the next thing you know, we have this mass of snowball and it’s awesome and everyone is just like cheering but you don’t get to see that till later. So all the hard work is in the beginning.” – Participant 25 Customers ignore useless features: “Ultimately, your customers, will eventually get rid of it on their own. It will become useless or your product will not be used.” – Participant 25 “We are always discovering new features, framing them, and prioritizing the next batch of work.” – Participant 44 “For me [Track 1] is a to-do list. I have a list of questions to answer that are prioritized by risk. Once I answer a question, I move onto the next one.” – Participant 51

has a reliable process for determining what to build. Other factors do influence product success such as market timing. “[Instead of just doing sensemaking, doing all of track 1 is more fun.] We got to do all the activities, and I have to say those are the kind of projects were I have more fun.” – Participant 52 Different starting points, same goal Wildly variant in starting points, trying to converge to the same point (Each project is different and the start of Track 1 is to get the project into a place where developers can work on it.) "Every [start to Track 1] is different, depending on the client team, depending on the problem space, depending on the location of the client and team.” – Participant 25

Fluctuation between Sensemaking and Context-Solution coevolution “There’s a fluid overlap between [Sensemaking] and [Context-solution coevolution]. The transition is subtle and the clients often want to know when it is happening.” – Participant 50 Purpose

Desiring fast feedback loops “When I start [Track 1], I want to know 1) What do we know. 2) What do we not know. 3) What is the most important to learn.” – Participant 51 Building empathy and understanding the user (Sensemaking) “The whole D&F team builds empathy for the user.” – Participant 41 “Something that helps with a DNF is building empathy for your user base.” – Participant 41 “Having gone through that research phase really helps with decision making moving forward. The more I know about my users early on, I rely on that information from a Discovery and Framing throughout the rest

of the [development] process. I’m always referencing my learnings. That’s why this is the thing we have to do. We have this foundation that helps us make faster decisions when we’re in the implementation phase” – Participant 41 Determine why people want to use the product (Sensemaking) “I like solving problems for people and so going out and seeing and understanding where they’re at and how they think about things. I like understanding people’s behavior and like why they do things that they do. I want to see application of the research I do. Discovery and Framing for me is the way to understand people and their problems and then like actually come up with a solution for it.” – Participant 41 Explore problems, solutions, challenge assumptions (Context-Solution Coevloution) Purpose of is to explore problems, solutions, and challenge assumptions “The actual real evaluation is if the product makes the money or actually brings the results that they’re looking for but we don’t get to see that until the end.” – Participant 25 Exploratory research understands business practices, landscape, impact of changes, for the purpose of identifying areas of opportunity DNF Purpose 1) prioritized problem and 2) prioritized solution: “We have prioritized problem and solution that help achieve the client’s goals. We know what’s going to be our MVP. The MVP is based on the scoped problem area and the solution we think that fits it.” – Participant 41 Example prioritizing problems: 1) not enough time for training, 2) many tools with copy and pasting, 3) matching asset to operation plan, Which problem space to focus on?, “There’s a bunch of problems, prioritize on the problem that we’re going to focus on. Then based on that, we’re going to come up with a lot of solutions and prioritize the one that we think is going to be the best for the prioritized problem.” – Participant 41 is not about finding what you set out to look for

“ is the bread and butter of design’s problem solving.” – Participant 43 Starting with a vision Reducing Risk (Context-Solution Coevolution) “[Track 1 is] just to reduce risk .” – Participant 25 Minimizing risk: “We got enough so that we felt confident like moving forward and felt we minimized enough risk.” – Participant 41 Design process reduces risk “I often see new learnings from user interviews and that we were wrong about something as very valuable.” – Participant 44 Purpose -> pain points vs opportunities

Looking for opportunities “We’re looking for opportunities but we’re focusing on the needs and the pains. The gaps are the opportunities. The disconnect between the software and the user is an opportunity.” – Participant 41 looks for areas of opportunity, not pain points “If the whole message is identify pain points and then from there to solve from it, I think then we are losing point of what the exploratory research was for” – Participant 25 Client focused on present pain point and uninterested in huge opportunities Design thinking is not about pain points Prefers not forcing the (interview) conversation to pain points Framing from areas of opportunity. “Reframe our mind-set and look at it from areas of opportunity, not just from pain points or insights” – Participant 25

Some offices zero in on pain points quickly while other offices start more broad The ability to revisit an interviewee relieves pressure Prefers prioritizing opportunities over prioritizing problems “I use the words ‘opportunity’ and ‘gap’ a lot and I notice a lot of designers I work with when I paired, they use ‘pain’ and ‘needs’ and I just don’t frame them that way. Of course, I’m focusing on the pains and need. Of course, for the ones that are focusing on the pains and needs see those as opportunities” – Participant 44 “Sometimes an opportunity doesn’t have a pain associated with it but a pain and a need is always an opportunity.” – Participant 44 Choosing between solving an immediate pain point or pursuing larger near-by opportunity An example D&F is opportunity to start thinking about other ways to measure success D&F explores new opportunities Scope

Limiting the focus of excludes potential business opportunities Management informing designer not to share product idea outside of area of scope “I presented it and said you have this area and they were just like no, no, no, our budget is for the forecasting tool” – Participant 25 Looking around area of focus: “I push a lot in DNF kick-offs to really define goals in the way that help us be a little more open ended about where we solve within.” – Participant 41 Area of focus helps: “It does help to have an area of focus. I’ve had a DNF that was so open ended that’s just understand this persona and we’re like well, this persona has so many things that they do like is there

anything in particular you want to focus on?” – Participant 41 Look around area of focus: “I would say like take that area of focus and step back like a little bit enough to not see the problem and like in the middle of it but see it at a slightly higher level so you can see like the surrounding things that are causing it. So you know what is really causing this issue.” – Participant 41 Researching something that client didn’t ask you to research feels risky Courage for suggesting client to look at another area Duration

“Pivots writing project proposals have approached me in the past asking "how long [do we start Track 1 before adding developers?]" Honestly, it's a hard question to answer specifically and seems like a stretch to prescribe a time period in a proposal. Being flexible in that a D&F could last a week or 3 weeks feels important. I'd also say D&F's can be exhausting - allowing for more breaks or down time would be helpful.” (survey response) “I'm really interested in the nuance of determining the length and scope of the D&F. If a project is pretty low risk, there is still some need for a D&F phase, but it doesn't have to be 6 weeks long. If you only need three weeks total to get the project started, is that still a D&F with approx a week and a half each of Discovery then Framing? I know we also have engagements called 'Design before development' but haven't seen a scenario where we actually just did Design only, building off of some existing research from the client.” (survey response)

When to skip

Always doing Track 1: “After the first release, you’re always doing these practices but at a more rapid pace.” – Participant 25 Skipping a DNF if they’ve done equivalent work: “If a client comes and they have a lot of good research, they have validated a problem and a solution and if their methods and our methods are similar, then you can just jump straight into the implementation.” – Participant 41 Keeping a DNF for all clients: “If there’s still a lot of unknowns and the problem is not totally validated then I would definitely recommend doing a DNF.” – Participant 41

Wireframes are not enough: “if a client had some other vendor create wireframes and that we were going to build them, it we would want to validate the solution.” – Participant 41 Avoiding just building existing wireframes. Validate. Antipattern: Skipping D&F. “The project did not have a Discovery and Framing. We thought the client designers had done some research about their persona. We kept having so many questions along the way and we tried to do it all through usability testing but at the end, we realized like a DNF would have been really helpful. We didn’t have enough information about the user to be making quick decision as we were going through implementation. Now they are starting another project and they are going to start with a DNF because they saw the value.” – Participant 41 Reducing risk. Risky to just start with existing wireframes: “it’s taking on risk to just start building something if we haven’t validated it.” – Participant 41 When to skip DNF: “our metric would be not whether the client has a clear understanding of the problem, the solution but whether they’ve actually done research or not. So if they brought research, we’ll skip the DNF. If they haven’t and they think they understand the problem and solution based on product experience, maybe we might still want to do a DNF.” – Participant 41 After the team has done the user persona and the team really understand their needs, and the team is implementing features, sometimes the next set of features the designer leverages existing research and understands what the user’s going to say or respond to the feature. The designer knows the answer. When this is not the case, the designer really does not know what the users are going to say about this feature, then the designer needs to get more data. (Participant observation confirmed by interviews.) Brownfield projects

“The whole point of upfront at day one is to figure out what is really driving people to use [an existing product] right now.” – Participant 25 “[For an existing product,] it’s like features and our micro mini ones and you can add onto it. It’s basically stitching on another thing or growing the idea that you’ve had already.” – Participant 25 "I would suggest to a customer or client for as a customer, you should make a new product. People’s

attention spans are so low now like the successful products nowadays are very specific." – Participant 25 Exploring existing tool: “On non-greenfield projects like spending more time exploring and understanding the existing tool is really important.” – Participant 41 “As a product matures, you tend to focus more on the validation and usability research with users. You are also utilizing data coming from your working software that's going into the hands of a high volume of users and/or getting continuous use in a real world environment. All these will yield new learnings that the designer must react to and ultimately generate stories in the backlog.” – Participant 56 Toolbox of Practices

“DNF is kind of having this toolbox and you know a lot of different methods and you pull the right one for the right situation depending on what you need in that moment. There are times that I’m like we have this interesting situation, let me go check out the guide and see if there’s like what’s the best way of approaching this type of problem.” – Participant 41 “DNF is having your toolbox of different ways of trying to frame the problem and solution.” – Participant 41 It would be really helpful to have a CORE activities list and a optional list that is based on the needs of the project. With guidance on why and how I should use these activities." (survey response) “I rely on principles over methods. Our principles are to work small, manage risk, and share knowledge. The designer's task is to build a body of knowledge by which the product team can feel confident they are discovering the reaction of real audiences in the real world. I feel this is shared less than methods and as such, paints a discussion on tools rather than outcomes. If we focus on outcomes, we can shape the tools to meet our needs.” (survey response) “The nature of the beast is to use the exercises as needed throughout, individually or collectively, to drive out uncertainty in any area whether that be pre or post-MVP” – Participant 45

Overall

Sensemaking

“It’s a trust game. The client is unsure about what will happen. I’m telling them to hold my hand, this will work out. The whole thing is all over the place. They have a hard time following why we do these activities in the [Sensemaking]. It gets better when we start framing a solution.” – Participant 50 “It’s supposed to be just an inquisitive escape of mind and I find that D&Fs are just constantly just like wondering like this is the phase. It’s just constantly wondering but we have activities to get us out of that like ideative, speculative questioning state and we have activities then to extract it and then we’re back to this and then we have some extraction and then we’re back to wondering like we’re always just wondering and questioning.” – Participant 44 “We are not all encompassing all knowing, omniscient beings. Even with software, we could be wrong.” – Participant 44 “Some clients are reluctant to do it because they haven’t done research but they think they know the answer, they think they know the users.” – Participant 44

Characteristic

Confusion to Clarity “With this type of research there is a lot of ambiguity… ” – Participant 25 Clarity to confusion: “I see it starting as a big tangled mess. I start unraveling the knots. I then understand the threads. I organize it into a ball. As a designer, I then can make it into a t-shirt or a sweater.” – Participant 46 Confusion to clarity: “Everything’s fuzzy and all of a sudden, it becomes more crystalized at the end and that’s really enjoyable.” – Participant 41 “I really like because it’s on a personal level. I get a lot of joy from when solving a puzzle or when something is like… usually when we enter a DNF, we’re like the problem is vague enough where there’s a lot of unknown and it’s really open ended and everything seems kind of messy and they get a lot of joy out of going through a process of being really methodical and then at the end, somehow even though in the middle of the process I feel like sometimes I question whether we’re going to get to some prioritized problems. Somehow, we always get there.” – Participant 41

“Being in awe of that ambiguity being comfortable with it. It’s just like that is true that I feel like D&F is the really wide part of the funnel and you’re just in it for a while so the funnel is really not very steep at the beginning and it’s just incrementally getting narrower but being okay with that and being in a state of wonder and being okay to ask the questions that you don’t know the answers to.” – Participant 44 Holding onto inevitable ambiguity until reconciling in a concrete solution “We say there’s inevitably ambiguity. ... You reconcile the ambiguity before you get to the point of a solution that’s concrete.” – Participant 25 Purpose

Intrinsic needs Understanding the intrinsic needs of the user Understanding why they think about X. Why is it so important? “So what’s the intrinsic need there?” – Participant 25 Purpose is to figure out unknowns: “[When the client] agreed to do a DNF indicates that they agree that research is important. There’s something that there’s some unknowns that we have to figure out.” – Participant 41 “No matter how well the client understood the problem space, I almost always adjusted something.” – Participant 43 “On a personal level, I get a lot of joy from when solving a puzzle” – Participant 41 Sets goals for the project “I hope their leadership realizes how important our research was because it really defined how… and when I say our, I mean the whole team. It really defined how and what direction we move and pivoted the

product to what was actually valuable to the customer, what gets surfaced right away, what doesn’t, what gets hidden, what can we forego.” – Participant 25 Enables quick decisions later in the project During implementation, make quick decisions about the user: “Without Discovery and Framing, we didn’t have enough information about the user to be making quick decision as we were going through implementation.” – Participant 41 Results used throughout project: “I use the results from DNF so much to make decisions about how I lay out a page and like the direction. There are so many ways to solve a problem, knowing more about my user helps me narrow the solution set significantly. Otherwise, I’m like sometimes taking a stab in the dark and doing like multiple rounds of usability because it’s not testing quite right whereas if I would have had more knowledge about the user, maybe I would honed in on the solution a lot more quickly. ” – Participant 41 Additional practices

Sensemaking -> Stakeholder Mapping

A few designers use Jobs to be Done (survey result) A few designers use Business Model Canvas (survey result) Several designers use Service Blueprint (survey result) Several designers do competitive analysis (survey result) Several designers do Success Criteria Generation (survey result) Many designers list assumptions (survey result) Several designers do story mapping during the dev inception (survey result) Many designers create a Design Brief (Insights, Personas, Flows, Mockups) (survey result) Most designers use Risks and Mitigations (survey result) Many designers use stakeholder mapping (survey result) Several designers create a persona ecosystem (survey result) We observed the team creating stakeholder maps to determine which client employees and end users to interview (participant observation) Client interviews: “Understanding like why does the client want to solve that problem is the most important

thing” – Participant 41 Sensemaking -> Interviewing

Techniques

Almost all designers do exploratory research / user interviews (survey result) Most designers interview stakeholders (survey result) Many designers do contextual inquiry (survey result) No designers do guerilla interviews (survey result) Most designers create learning goals for user research (survey result) Most designers create semi-structured interview script survey result) Most designers create a topic map (popcorn diagram) as an unstructured interview script(survey result) Asking the interviewee to sketch: “If you were to draw your ideal tool for doing your job, what would that look like? … Tell me why you did that and then they’ll be like oh it’s because I need to do this thing. It sparks conversation.” – Participant 41 “During the interview, I’ll use topic maps for my guerilla interviews. Most of the time I have a proper guide.” – Participant 52 Aligning user and business needs (D&F)

Purpose User interviews is to reduce risk of success of product Quantitative data doesn’t tell why: ​“You have data but you don’t understand why these calls are coming

in. That’s why we do open-ended research. You have the quantitative data. We need to understand why. You have the symptoms. I want to understand the source of the problem.”​ – Participant 25 Understand the user’s perspective (See Obstacles for issues) (See Track 1 - Sensemaking for more codes)

“[I’m not asking for implementation details.] I want to understand how they think about their bills. Why are the bills so important?” – Participant 25

“So even though you can observe somebody and they’re doing a particular thing, I want to know why they’re doing it that way.” – Participant 41

Getting client stakeholders to listen to user’s concerns: “If only the stakeholders could just sit with the users and listen to them, the stakeholders would realize that the users do not care about all these ideas that the stakeholders have.”​ – Participant 25 “User’s needs can change.” – Participant 44 Open ended vs closed

Using open ended research to understand persona’s pain point Using directed questions. “We’re doing research to understand like this part of your process because we want to build a tool that helps you do whatever.” – Participant 41

Ask very wide open questions Allowing for emergence of issues during the Design and Framing How many?

“​So the goal, I always told them it’s four to six [interviews] for validation.”​ – Participant 25 5 interviews per persona: “I’ve had DNFs were there are multiple personas so I would say like five for one persona is good.” – Participant 41

Other

“It can be tiring but I really enjoy research personally.” – Participant 41 “I also really enjoy talking to people and doing the interviews and going out in the field and seeing people.” – Participant 41 “We just finished a project where we did only discovery for 3 weeks. For CLIENT identity application the team gave a recommendation for how to handle the upcoming EU regulations for data privacy. The team

gave the client a [sensemaking] experience report with all the findings, interview insights, etc. In that case nothing was framed and non of the framing activities were done, no sketches or prototypes. That's what I meant that depends of the problem we are trying to solve we use what we need from the process.” – Participant 52 “A lot of times, especially if it’s a client that’s not user-centered, they are very frustrated to go and learn about their user or their demographic. They find our user interviews (and thus our synthesis) as very tedious. But they love it when we’re going to a subject matter expert or a product owner or some sort of champion of that product and interviewing them because they feel like it’s basically just resuscitating requirements in some way. I think they feel more comfortable with that.” – Participant 44 “A lot of clients come in feeling they know everything they need to know about their users, and the way I tried to mitigate that is... If they do know a lot about their users, I ask them for the artifacts and the interviews and the analysis that they have and I tried to consume it and recite it to them or incorporate it into what we’re learning so that they feel like we’re leveraging what they do. So I try to mitigate it that way but they tend to feel they know everything there is and everything’s static. Users are like this three months ago so it’s got to be the same, we did the research, we have the focus groups, we have the demographics. And honestly sometimes, they’re right. Sometimes, they do know everything they need to know about their users. It’s more of how are you as a client going to share that knowledge with me so that’s a challenge and I try to mitigate that feeling of frustration with bridging that gap.” – Participant 44 Clients are surprised to learning new things from the user interviews “Always, always and we try really hard not to be like an “I told you so” when we challenge something and then we learn it in an interview from the horse’s mouth.” – Participant 44 “I often see new learnings from user interviews and that we were wrong about something as very valuable.” – Participant 44 It is a good thing when we learn that we were either correct or wrong about our assumptions Sensemaking -> Persona Modeling

Most designers create proto personas (survey result) Many designers create molecule maps (Proto-Persona, Initial Problem, Initial Solution) (survey result)

“Later on, [the client] said we didn’t realize how important the users were in the beginning.” Make personas for backend systems: “You can make a persona for anything… At the end of the day, there are people managing it, using the system.” – Participant 25 “Success is creating clear personas, did we have some insights to pull from that, are there findings from those insights that we can then draw into areas of opportunity” – Participant 25 “I keep refining my personas through the whole time.” – Participant 49 “I like when we come up with a persona not necessarily the characteristics but the behaviors and the needs and the problems, those always helped me.” – Participant 44 Techniques Sensemaking -> Affinity Mapping

Almost all designers cluster post-it notes / create affinity maps (survey result) Most designers distill design insights (survey result) Many designers distill and refine personas from affinity maps (survey result) Almost all designers prioritize the problems to solve from affinity maps (survey result) Almost all designers prioritize solutions (survey result) Many designers revise the value proposition (survey result) Many designers create scenarios (survey result)

Uncertainty to certainty

Uncertainty resolved by synthesis: “is anything going to come out of this? it’s not until we start organizing the information and synthesizing that we see the patterns” – Participant 41 Uncertainty: “Are we really going to get anywhere with this. Somehow, it always works out.” – Participant 41 The uncertainty lasts about two weeks Creating a feature list

Deriving features from insights and user needs (D&F) Comparing how different people ask for a feature Triangulating a feature by looking at multiple ways people have suggested a feature Anti-pattern: customer to backlog “An anti pattern is observation to backlog. One person did one thing that we observed once, and that tumbled into the backlog. This creates a Frankenstein amalgam of features. Microsoft products exhibit this. Instead, we want insight-synthesized research. Patterns emerged from the synthesis of data. Insights earned their way.” – Participant 43

“My patterns become learnings. If I have a ton of learnings, I pull out my key learnings. For me the word “insight” is a magic moment that informs a particular direction in an opportunity.” – Participant 46 "One of the tabs was newsfeed and no users wanted newsfeed. … the only people wanted the newsfeed was marketing." – Participant 25 “The board member wanted it and [the team build it even though users didn’t want it]… you tell them let’s put some analytics in and track it. Let’s see how many people are actually… what is the conversion here, what is the goal.” – Participant 25 Anti-pattern:“The idea of aggregating and sorting is prescriptive. Here are the problems and here are the solutions. You are not exploring solutions. You are not exploring problems. You are not challenging assumptions.” – Participant 46 “Oddly enough for someone that works in a digital space, I enjoy working in analogue and like with my hands and moving around and on whiteboards and I feel like DNF provides a lot of that.” – Participant 41 Start synthesis promptly after interviews: “after we’ve done a lot of interviews, we haven’t had time to synthesize but there’s so much data” – Participant 41

Synthesis is a phase, with affinity mapping is a method you choose to synthesize. "Discovery - observations / interviews, Synthesis affinity mapping, Solution generation, ⇒ Prioritized problem and prioritized solution" Getting more data after synthesis: “If you synthesize and feel like you still have a lot of gaps in knowledge then I think it is worth going back to your users.” – Participant 41 Rarely needs to get more interviews. One iteration enough. Discovering that the market is more like one persona than the identified persona

Two opposing things present opportunity: “Whenever you have a point of conflict where there’s like or maybe not even just a point of conflict, it’s like a point of two opposing things like for some reason, they’re not in congruence and you’re not sure why, there’s always opportunity there.”​ – Participant 25 After synthesis do 2x2 or prioritization matrix for prioritizing problems Overall

Creating a clear product vision

Context-Solution Coevolution Reconciling multiple opinions to create one clear vision “"[Some clients think] We do like a PowerPoint slide and this just kind of shows up and this wireframe just magically appear." – Participant 25

Focus products on doing one thing well: “Enterprise clients just, they’re like no we need this complex link, no actually what you need are a little [product], it sounds crazy but like microservices like we do here that are integrating together and working together under a suite of products that you have like that’s kind of how I look at it .”​ – Participant 25 Need to create a product enforces constraints: “Because we have to build a product. So there’s that constraint. Ultimately, we need to sell something and like I’ve kind of flipped my brain to be less open to

the idea of like we won’t have a product in the end so that we have products coming in and making a sale. So that was a new inflection point that I’ve had recently in the last year.”​ – Participant 25 Me: “In the double diamond diagram, we show divergence followed by convergence. If we are flowing back and forth, what is doing on?” Participant 50: “We are iterating on the solution generation and prioritization.” “[Context-solution coevolution] is taking all the problems and the pains and framing them in a way to produce a solution….This part is hard.” – Participant 44 “We’re focusing the solutions on the product because there are million solutions to million things….it’s blowing it open in terms of solutions, funnelling it down in terms of problems and framing it so it relates to our goal.” – Participant 44 “And the thing is this designers already probably have the thing in their head including product managers like we already know what we’re going to do. We’re already starting to frame what our solutions are going to look like. This is just to keep us honest. This is to keep us in check and this is to develop a sense of empathy with the team and alignment on ‘yeah, this is why we’re solving this.’” – Participant 44 Getting the business opportunity to the right stakeholders is hard Difficult to get findings to the right people: “Am I actually giving the findings that are actually super valuable to the right people? The answer is no. The true answer is no.” – Participant 25 Finding business value

“We often struggle with finding/defining business value.” (survey response)

Clearly aligning features with revenue “What is the question we are trying to solve? Where is the gap that the product fits?” – Participant 43 “Leveraging research to substantiate why they need to do this project.” – Participant 25

Using two by two for balancing business needs and user needs

Using 2x2 with business value versus user value to have client prioritize Using prioritization for balancing business needs and user needs DNF explores if there is something there worth building “The solutions go really wide. They could be technical solution, they could be backend solutions, they could be front end or user facing solutions. They could be service or process solutions. They could be marketing solutions but that’s what’s so fun like the solutions blow up but now these are all solutions for the core problems that we framed.” – Participant 44 Finding unexpected opportunities

Communicating an idea isn’t going to work by conveying there is a higher growth opportunity with in a different product area. “How do I convey to them that over here is not a great area but we may have opportunity here and that’s the better route to go?” “They wanted a product that [tracks their inventory] but really when we started doing research, there was a huge financial backlog that needed to get better organized and it could just have been fixed by actually teaching their people how to use Excel really well and really using it in an efficient way.”​ – Participant 25

Validation

“We want to validate why we’re making the decisions we’re making.” – Participant 25

"We have to make all our decisions objective, we have to make it clear to them that why we’re moving in a decision."​ – Participant 25 “Framing creates possible solutions and validates them.” – Participant 43 Showing evidence of results. “The D&F provides a framework and opportunity to show evidence.” – Participant 46 Context-Solution Coevolution -> How might we

Many designers use How might we (survey result) A few designers use Lateral Thinking Exercises (survey result)

A few designers use Rolestorming (survey result) Why-How Laddering – “The best place to be is somewhere in the middle where it’s a little defined but vague enough where there’s a lot of opportunity in that space but not so narrow that you’re like have one thing that you can do.” – Participant 41 “I do that (laddering) in my own head during the DNF of defining the problem in that like sweet spot.” – Participant 41 “You have all of these user needs and all of these persona needs and the brainstorming or solutionizing sessions gets us to start thinking about solutions and I start to feel like more concrete and then there’s something we call rolestorming. It’s when you’re thinking of analogous or complementary like businesses and how they solve for their problems and what are their value props to their users and what can we learn from that to apply to our problems.” – Participant 44 “I’ve tried rolestorming three times. One time, it was successful all around. Everybody loved it. One time, it was successful with just the designers and everybody else hated it and the third time, it was a disaster.” – Participant 44 “Rolestorming gets people out of their box and get some to think about how successful [companies] cater their business solutions to their user need.” – Participant 44 Context-Solution Coevolution -> Sketching

Almost all designers use design studio (survey result) Almost all designers use sketching (survey result) Group sketching elicits multiple perspectives and helps align team. “We will all sketch out our ideas, put them up, and then talk about what we came up with. That way, we’re all moving in the same direction.” – Participant 41 “We’ll brainstorm various solutions to the problem then there’s the prioritization of the solution and that usually I do a two by two.” – Participant 41

Mapping user’s mental model to presentation of information Separating information that is not linked Building wireframes. “When I’m building a wireframe, I need to know what data the users are using. There’s information I need that is relevant to the flow. I also care a lot about the mental model or the approach of the users taking to their workflow. I’m not going to design it exactly in the way that they do it. I’m going to design it to optimize to what the goal is and if I don’t know the why then I’m missing on the goal there. That is the hardest part.” – Participant 41

Scenario Walkthrough Part 1: “the scenario walkthroughs I think are really important and that I think people tend to… we are problem solvers and we try to jump straight to coming up with solutions before we, I think, fully understand a problem and I think the scenario walkthroughs I’ve been using a lot of because the point is not to define what is the interface. There are lots of ways to have an interaction that achieves the same goal.” – Participant 41 Scenario Walkthrough Part 2: “you’re talking through what does the user need right now, what do they do as a result of that and then how do they feel about that interaction and they continue to walkthrough that story through a whole flow without defining the interface and then we have a much better sense of like after each step in the flow like why do they want to do it and then what’s the next thing that they need based on how they feel and if we can get through that then with stakeholders like if we all agree on what we’re trying to achieve through that writing that story then I can take that and like create wireframes out of that and think of like what’s the best way of achieving this thing and that I found really helpful.” – Participant 41 “Regarding solution prioritization. This is what happens during a design studio. We dot vote on items. Solution prioritization is not a separate phase.” – Participant 50 “The sketching exercises takes us about 2 to 5 hours. We are divergent with many different ideas, and then we keep doing it until convergence.” – Participant 51 “Those first design studios are really important for [context-solution coevolution], the solutionizing phase.

The personas are important, scenarios become important, scenarios putting the persona in a day in the life are important.” – Participant 44 “I actually outline a design studio by the agenda so that everybody like is ready to get going. So it’s like let’s review our persona, let me give you a user objective right now that we need to solve for and let me put you in a scenario for that persona who’s trying to achieve that objective and that is informed by all of the user testing, the user interviews, all of the stakeholder interviews. I make sure that my scenario has data points that we have leveraged from user interviews so that they could kind of see the connection. When you put the user in that scenario and you give them a clear objective, now the clients in the room can really focus. It’s really effective and I really try to make them comfortable. I’m not concerned with fidelity. I’m not concerned with illustration. I’m concerned with boxes, arrows, copy, stick figures, house models, whatever they think solves this problem in a visual way. So that will help disarm them and keep them comfortable and solutionizing. Then with my team like my product manager then can abstract those into solutions. We evaluate them collaboratively. Nobody gives any context on their illustrations and their sketching and people just look at them. We do dot-voting. [A vote means] wow, this looks interesting, or I’d like to hear more about this, or I like this solution. We get people who voted to talk about why they voted before anybody explains anything. So the image can speak for itself and then the person who sketched it can speak through the person who voted for it. I’m only looking at the ones that get the most votes so it’s like a heat map and you can prioritize the most interesting solutions and then that person then gets to… after the voters talk about why they voted for it then the sketcher can talk about what they did. ‘that was exactly what I intended’ or ‘yeah that they got what I sketched, that was my solution, the reason I did this is because blah, blah, blah’ and you just kind of go through them that way. [The process is iterative until converging on one set of wireframes or mockups.]” – Participant 44 “Often times, the developers give us the most interesting interesting points of view. They flip the designer’s model with their sketching.” – Participant 44 Context-Solution Coevolution

Almost all designers do usability testing (survey result)

-> Usability Testing “We’ll go back to the users to validate our approach to solution. We’ll get a little more detail about the problem. It’s not a whole exploration [like we did initially.]” – Participant 41 Usability testing addresses solution concerns Keep usability testing a product “If you test it and you keep testing it, you’re going to reduce the risk. You’re going to find out early on that something is not right. All of a sudden, people don’t like this and then you can actually ask why. What is it about this that you’re not as interested in it anymore and something like that like tell me more and you’ll find that there is this new competitor that comes in doing something cool.” – Participant 25 “We’re going to put the billing information and the usage information in this particular project on the front page. We knew that. We know but whether or not like if the way they think about billing and usage is interlinked, that’s the difference between two separate pages and one page ... But you don’t find that on until you talk to someone and you go, how are you currently monitoring your employees when it comes to managing your telecom services. It is a very wide open question.” – Participant 25

“Newsfeed just gets put in the prioritization. Here’s the thing though, when you start building it or when we start wireframing it, we’re going to go show it to users and then what I always told our clients is when we show it to users and you start getting negative feedback on it, can I have your word of whether or not you were still going to be adamant to put it in because merely all we’re doing with these user interviews is to reduce risk of the success of the product.”​ – Participant 25 Context-Solution Coevolution -> User Story Writing

Producing a backlog for first couple weeks of development Break into initial stories for the first day into a backlog (inception) Create stories (PM, Stories)

Backlog Attributes

Stories are features, bugs, chores (participant observation)

PMs shouldn’t described implementation details PMs need to understand the product and user needs, not the code base Create stories for performance quality attribute in tracker Writing quality attributes from a user perspective Quality attributes are still requirements and can be in tracker Create stories for usability quality attribute in tracker Each story should add business value / only adding high business value features to the tool “developer-ready” stories

Verifying that each story is ready to be worked on Identifying clear acceptance criteria for a story necessary for an IPM Attaching design to a story Picking up a story, the pair looks at the acceptance criteria Providing a motivation for a story makes it clear when a story is missing one and shouldn’t be a feature. Identifying motivation and acceptance criteria PMs should not described implementation details

Boundary Spanning

Designers enabling developers

“[Once developers are writing code], much of designers time is spent on enabling the developers, answering design questions” – Participant 32 “We ask the client to let us prioritize and discover the what to work on *first* (that take longer) as well as define what the MVP will be. After inception (which marks the end of the D&F period), we essentially continue to discover the next batch of features, a D&F on a much smaller and faster scale.” – Participant 44 Track 1 needs to produce a product since there are engineers allocated to work on it. (Participant observation confirmed by interviews.)

Boundary Spanning -> Backlog Grooming

Building the next most important thing

Delivering what is valuable next Delivering best value at each moment No estimates focuses on momentary value of what needs to be done next No Estimates helps you focus on what the next most valuable thing (IPM) Building the next most important thing next Building the right thing first Building the most important business value first, then iterate

Tending a healthy backlog

Reviewing backlog Fixing an unhealthy backlog Tending the backlog More about keeping the backlog healthy than setting a goal for Friday

Keeping the team in flow means having a “healthy” backlog Organizing backlog in tracker PM owns the backlog PM owns prioritizing the stories Changing “tasks” into “stories” with acceptance criteria Prioritize stories (PM, Stories) Rearranging backlog Ruthlessly Prioritized Backlog

Misaligned priorities is not ideal Ruthless prioritization Ruthlessly prioritizing the backlog (PM) Prioritizing backlog Prioritizing stories after they are estimated Aligning stakeholders Prioritizing backlog (core principle) Providing the PM with the ability to steer prioritization (Backlog) Prioritizing features (core practice)

Creating and prioritizing a backlog (PM) Using Dump and Sort and 2x2 to prioritize features 2x2 of development complexity (time) and value (impact) for prioritizing features Dealing with prioritization block for feature prioritization (Prioritizing features) Comparing feature importance to establish priority order between them (Prioritizing features) Prioritizing product goals to help prioritize features. Using “iron triangle” to help client prioritize features (Prioritizing features) Having fun with feature prioritization (prioritizing features) Cutting down a thick backlog Antipattern – “Why are you prioritizing this over that? With the [user research]I know why it is fundamental to success of the project.” – Participant 46 “There’s always like an architect telling you what the requirements need to be… We start prioritizing them and we think of them in buckets of features and [possible] experiments. That also throws them off. It is an exercise in ambiguity and that’s tough for them too.” – Participant 44 Features into epics

Epics Using Epic label for sequence of stories that build a feature (participant observation) Releases are a set of a few epics An Epic is 10 to 20 stories

Epics are things that you can start and finish, whereas tracks are longer lived tag structures Using empty epics to track long term vision Epics represent marketable work Stories can be added to an epic at anytime Prioritizing epics Tracking features by epic Grouping stories into epics Features into stories

Feedback on decomposing features into stories (IPM) User stories that communicate product vision Layering stories to accomplish a feature

Simple “sized” stories

Each story should bring business value Understanding complexity of the stories Using points to communicate complexity Points represent complexity Points aren’t time Estimating stories

Estimating in points, not time Breaking down things Making things granular Bring granularity down Big stories make people nervous Breaking stories into small pieces to enable pair rotation Breaking stories into small pieces to enable incremental delivery Incremental in adding stories to accomplish a feature Pointing useful for PM for feedback on writing similar sized stories Describing a tiny interaction between user and software Refining and verifying that user stories are ready for pointing Estimating for granularity of story The value of pointing is to know when a story is too big Pointing stories in an IPM Pointing as many stories as possible during an IPM Prefers stories to be self contained and does not like too granular of a story Prefers stories to be self contained and implement a piece of functionality completely

Too-small A story that creates an empty page creates negative business value Enough work

Making sure there is enough work in the backlog Pointing at most two weeks worth of work Pointing too far in advance means that the team forgets the context and discussion

Blocking stories

Blocking a story Discovering assumptions may block a story

Deleting stories

Deleting stories Deleting stories from the icebox when not in scope for a project Deleting things in the icebox and it will come back if it is important

Boundary Spanning -> Accepting stories

Accepting, Rejecting stories

Accepting stories Marking a story as delivered when it is deployed to staging, non-production website or in a build Done means you see it work in staging environment Acting as the persona in accepting testing Rejecting Stories Adding nitpicky label to a rejected story Rejecting a story versus filing a bug

Rejecting stories Boundary Spanning -> Story Showcasing

Understanding when product is releasing Refining and verifying that user stories are ready for pointing PM owns the IPM Balancing what you want versus what is possible

Coding

Working on stories (dev) (See previous research studies) Preconceiving problems

Obstacles

Lack of a market fit: “There is another problem of market fit. What is the viability of this? The non-willingness of our clients to be lean and test that approach is a problem.” – Participant 44 Antipattern: “they told us to go and look grant life cycle management so we went and looked at life cycle grant management and then ultimately, we created a prototype for a life cycle grant managing” – Participant 25 Misaligned expectations to project goals Re-aligning expectations to way of work and Track 1 outcomes Feeling down when project expectations not aligned to team’s goals “In the start-up community, there is sort of this idea that if there is not a market that you’re better off returning the money to your investors.” – Participant 25 Preconceived ideas for projects may not be a pain point or may be the result of something bigger causing

the identified pain (symptom of larger problem) Client focused on present pain point and uninterested in huge opportunities Ignoring user feedback “They give us some people to interview, we interview them. They did not want the product. They told us, ‘We interviewed the wrong people.’ Well, ‘what people do you think we should interview?’ and they would give us another set of people. We did this a few times. In the end, we decided, ‘we are just going to build this.’” – Participant 25 “We were talking to people and they were not really interested in this idea. There wasn’t a right fit with the persona the client was imagining. He didn’t really care. ‘I just want you to build this thing for me.’ So we pointed out the risk he’s taking on.” – Participant 41 If we are going to ignore the feedback from user interviews, then why bother with them at all. Unable to communicate what the client really needs when it isn’t what they want to hear Wondering how to reveal no one wants the product to the client “I get frustrated when we can’t do research because that’s literally what I’m doing. I’m going to have to pick things from the sky and then who gets in trouble?” – Participant 25

When client ignores user feedback, build it anyways. Ignoring user feedback “Some clients are reluctant to do it because they haven’t done research but they think they know the answer, they think they know the users.” – Participant 44 Preconceiving solutions

“The stakeholders can assume that they know what the problem is and what the solution is before we’ve really had time to process what we’ve learned. We’ll start interviewing. People are saying “I know it! We

don’t have to keep doing this research” and I’m saying “no, no, we have only talked to two people, we don’t know what’s there yet, we don’t know if that’s the solution, let’s think a lot of things.” ” – Participant 41 “The client [thinks] they know what the solution is. ‘I know the answers so why are you still asking the question?, I know the user’s need X, Y and Z.’” – Participant 44 “The client will say, ‘The users don’t know what they want. This is a whole new thing. They have no idea. I’ve already solved the problem for them. I am the user.’” – Participant 44 “The client says “We know what we want, we are going ahead and build it anyways.”” – Participant 43 “We could have just built them that product but their end users would not have been happy with it.” – Participant 25 “One of my clients said, “I’m going to build it anyways.” This client cancelled the project and went with internal resources to build it.” – Participant 52 "Our clients are very solution minded and solution focused. They don’t want to be in the problem space. The way we ask our [interview] questions [to the users] require us to be in the problem space and they don’t like that. It’s not a natural state to be in.” – Participant 44 Users not valuing product Need DNF when project does not have enough research to back what is needed to be built The team dislikes building products that the users do not want “I’ve been on projects where we’re building something that no one wants” – Participant 32 Designers disliking this: “we could have just built them that product but their end users would not have been happy with it.” – Participant 25

“But as an engineer, it was sort of not a fun project either because you’re like well no one’s going to want this thing.” – Participant 32 “Exactly, so I was just like… so that was my lesson. It was just kind of like okay like this isn’t my product, this isn’t my money.” – Participant 25 Allocated budgets focus teams on preconceived solution to the problem and excludes additional opportunities. “We’re trying to get a small thing to test to have an MVP, but our clients pervert MVP by wanting more features. It’s always got to have more features.” – Participant 44 Expecting a prototype for the D&F process is an antipattern “it’s like we’re going to give you a prototype. It’s like that’s very frustrating when that’s the outcome because it’s like it’s not the purpose of a D&F” – Participant 25 D&F focus on creating a prototype excludes identifying other opportunities D&F is more than creating a prototype Just build the product (Similar to ignore user feedback) Clients wanted to bypass the process and created wireframes without understanding user needs "I could have just listen to their leadership and just create wireframes. [But that was not in the product’s best interest.]" – Participant 25 “[For one client], we should have just gotten into execution mode and assumed the risk that he had accumulated with the way he was solutionizing for his users” – Participant 44 Premature Convergence

“I used to go super wide on solutions. But experience taught me that it is not worth it. Why go through all that work when the results are not desired. Our goal is to ship solid software (quickly).” – Participant 51

“In Theory, we want to go wide, but in practice we build them a website.” – Participant 51 “The sketching exercises takes us about 2 to 5 hours. We are divergent with many different ideas, and then we keep doing it until convergence.” – Participant 51 Dealing with Ambiguity

“How open ended and exploratory it is sometimes can be a little uncomfortable for people. The amount of unknown is very uncomfortable sometimes.” – Participant 41 Threatening to leave the engagement Needing consistency of feelings about individual contributions when project expectations are fluctuating Having tangible idea of output of open-ended process is stressful to participants “This tangible idea of what’s going to be is extremely stressful because it may not actually be what’s right for them” – Participant 25 “Some struggle with uncertainty, it is all over the place. You don’t know anything.” – Participant 52 “[To help with the ambiguity, I] have an agenda for the week broken down day by day. Have a spot for tracking questions. Every morning review agenda and answer questions.” – Participant 52 Could we normalize the ambiguity somehow with the client? Even setting the expectation that there is ambiguity, track 1 is still painful. “Pivots who are very uncomfortable with that ambiguity and it makes it hard on the team” – Participant 25 “Most of the time when clients are upset is because we haven’t managed the ambiguity well” – Participant 25 “It makes them so uncomfortable that they start doubting that they’re even moving in the right direction” – Participant 25 “There’s always like an architect telling you what the requirements need to be [who has not done user

research.” – Participant 44 “PMs really struggled with [ambiguity] the most. They are so used to having a goal and like let’s work towards that goal” – Participant 25 Countering the uncomfortability of ambiguity by introducing daily exercises that make us feel productive “We have to be comfortable in the ambiguity in the questioning phase” – Participant 44 Nice to reduce Ambiguity to Clarity feedback loop Duration of ambiguity makes dealing with ambiguity hard: “PMs like oh this dragged on too long, oh this took too long or like blah, blah, blah and I sit back and I looked at them like no I mean we had to do that like I wouldn’t say it dragged on long but yeah if you were measuring it against, you anticipated as being here and we had to go here then yeah like it went long” – Participant 25 Disconnect: success is comfort of client, but D&F make the client uncomfortability. Perseverance even when the rest of the team can’t see an answer: “I kept getting feedback of ‘there is nothing to solve’ and there’s like alot stuff that we just need to sit down and decide like where this is worth solving like are there enough nuggets here. I just designed two different mockups” – Participant 25 Dealing with ambiguity: “you have to hold someone’s hand, trudge through the mud together because to get to the other side, they’re probably having a wave of emotions the first time around.” – Participant 25 Emotional alignment of team. “I was kind of like in a different mind-set because I always felt like I was in opposition of opinion in that project which kind of sucked.” – Participant 25 (She was optimistic, team was pessimistic about finding a solution.) “So the clients usually in my experience are always frustrated with the D&Fs because it’s really uncomfortable for them to feel like they have a solution and then to come to us and then start from scratch again which is often the case. Very rarely is it such that… I haven’t been on many projects where they appreciated us framing their problems. So the things that helped the client like… synthesis definitely

doesn’t help. Synthesis is very painful for a client. When we are solutionizing and starting to apply them to the persona’s needs, I think they really appreciate that. I think they also appreciate stakeholder interviews more than user interviews.”– Participant 44 “User interviews tend to be a process that they [eventually] start to get comfortable in. [At first,] it’s like the top of the curve: the complexity and the frustration is high, high, high and then they plateau when we’re done with synthesis and they see us turn it into a persona and then when we start user interviews again, it’s sort of calming. Not as [stressful] much as those beginning interviews.” – Participant 44 Clients happy when teams start building stuff: “When they start to see sketching, when they start to see solutions to the problems in the form of either verbally stating the feature like in a list and a 2x2 and especially when we start sketching and then just like they love to see developers like come out of their technical D&Fs and start coding. They love to see designers like just in execution mode and the higher fidelity the better.” – Participant 44 “When I first started, I was like “what the hell is a DnF?” I was really scared. Now I have been through it many times.” – Participant 52 Creating just a prototype disappoints the product designer Time Pressure

Not having enough time to do a DNF Pragmatically focusing on target personas (limited time requires focused interviewing) Taking too long. “Sometimes people feel the anxiety of like is this taking too long, let’s start coming up with the solution.”– Participant 41 Designer has pressure Requiring patience in the DNF “​My client designer actually took a lot of heat.”​ – Participant 25

“[Designers] have to be like get politically savvy and have a lot of resilience.” – Participant 25 “if you have too much emotional attachment to what you were doing, it can be extremely exhausting and it will burn you out.” – Participant 25 No time to reflect after D&F, entire teams moves straight into building the product Realizing the need for more reflection time “I’ve been doing the motions not questioning them too much and then now I think I’ve hit a point where like I do the motions without thinking and now I’m questioning them which is good but then I’m going, “wait, why am I doing these things?” I got told to do these things, This is what we do here. This is probably it. I drank the Kool-Aid. It’s me trying to do better. I could have done it way better.” – Participant 25 Technique for too long: “Usually, it’s easy to temper that with saying we don’t have that much data yet, let’s make sure that we talked to enough people.” Technique for too long. “Talk about the risks of if we jumped to a solution too soon then we might start building something and then scrap everything and that’s a waste of time”– Participant 41 Blocking access to users

Preventing collaboration with other teams Unable to create a persona Using call center volume data to drive product needs "We’re not allowed to collaborate with that team." – Participant 25 “I couldn’t even create a persona for the customer service agent because we weren’t allowed to talk to them.” – Participant 25 Enterprise clients not participating in D&F

“I was not able to observe my users in person or meet them because there were security clearance issues. Getting the amount of detail that we would usually get in a DNF was difficult. We eventually resolved that by sending a team of just training a team of military people to conduct field research. Training them was tough. Sometimes, we got more detail than was necessary and sometimes, it was too vague. We got enough so that we felt confident like moving forward and felt we minimized enough risk. When we did our first round of usability tests, we had a lot of unknowns.” – Participant 41 Unable to acquire needed information. Redesigning a tool that the designer can not see: “We were redesigning a tool that was on a secret network and putting it on a non-secret network. We couldn’t look at a monitor that was on the secret network, so we couldn’t see the tool. Later we learned they could take screenshots and then declassify the screenshots.” – Participant 41