Skip to content

About this Episode

Watch the video of this episode: https://youtu.be/NPIjuY_0HYU
Support the show: https://www.buymeacoffee.com/datascienceharp
Resources:

http://ecai2020.eu/papers/348_paper.pdf
https://arxiv.org/pdf/1912.10564.pdf
https://cds.nyu.edu/wp-content/uploads/2019/06/RDS_TentativeSyllabus.pdf
https://dagshub.com/
https://discord.gg/ngNdE5Tvzy
https://diversity.google/annual-report/
https://hal.inria.fr/hal-01522418/document
https://insights.stackoverflow.com/survey/2021#salary-comp-total
https://www.kaggle.com/discussion
https://www.linkedin.com/feed/update/urn:li:activity:6889576309601640448/
https://www.linkedin.com/in/reid-blackman-ph-d-0338a794/
https://www.microsoft.com/en-us/ai/responsible-ai-resources

Don't forget to register for regular office hours by The Artists of Data Science: http://bit.ly/adsoh
Register for Sunday Sessions here: http://bit.ly/comet-ml-oh
Listen to the latest episode: https://player.fireside.fm/v2/eac-KT9/latest?theme=dark

The Artists of Data Science Social links:
YouTube: https://www.youtube.com/c/HarpreetSahotaTheArtistsofDataScience
Instagram: https://www.instagram.com/datascienceharp
Facebook https://facebook.com/TheArtistsOfDataScience
Twitter: https://twitter.com/datascienceharp

Transcript

HH65-21-01-2022_mixdown.mp3

Harpreet 00:09

What's up, everybody, welcome, welcome to the artists, the Data science, happy hour. It is a Friday, January 20. First, it is Data Science. Happy hour number sixty five. I feel like every few weeks now, we just hit another milestone milestone after milestone after milestone. Super excited to have all these guys here. Hopefully, you guys got a chance to tune in to the podcast episode that was released today with Dr. Joe Perez. Talked about, talked about a lot of stuff, man. So hopefully guys tune in and check it out. Talk about how to bring your

Speaker2 00:40

Story to life,

Harpreet 00:42

Your Data story to life and make it actionable. So definitely check that out. Hopefully, you AIs got a chance to tune into the comet office hours that we did on Wednesday. This week, we covered pretty much the importance of baseline models, did a quick presentation about that, as well as the accompanying blog post, and had a awesome community member come in and share her experience

Read Full Transcript Hide Full Transcript

Speaker2 01:06

In,

Harpreet 01:09

You know what it's like for her to to build models in her workflow and her process. So it's always fun to see how different folks do different things to hopefully get a chance to tune into that. It's available on my YouTube, also on the Comet YouTube as well, so definitely check that out. Shout out to everybody

Speaker2 01:24

In the building.

Harpreet 01:26

Dave Langurs Back to Earth, Darth Linger. It's good to see you again, Dave. It's been quite some time. Dave, what's going on, Russell? What's going on? Kanji is in the building. Eric Sims, Matt Damon and of course, last but

Speaker2 01:39

Not least, the legendary

Harpreet 01:40

Ben Vicious. The super excited. Have all the guys here, man. It feels like we get the old gang back together, man. Just give me flashbacks to a, you know, well over a year ago, man, get to see all you guys here. So. Yes. Let me open this question, man. I've been trying to find places

Speaker2 02:00

To get

Harpreet 02:01

More information about data science and machine learning. I know all of us tend to hang out on LinkedIn. That's how we all know each other. That's where we usually go for for knowledge and information. But where else is

Speaker2 02:12

There

Harpreet 02:13

That we can go to ask questions? Maybe we want to ask questions pseudonymous or maybe ask questions, you know, in a forum where it's not as as public, so to speak. What are some other awesome communities that that, you

Speaker2 02:30

Know, I mean

Harpreet 02:31

About community. I mean, just like other places like whether it's, you know, different

Speaker2 02:35

Communities, whether it's

Harpreet 02:36

Different websites, Stack Overflow, Reddit, Quora, things like that. Where do you guys go for four or four awesome information and help when you need it? They linger. Let's listen. Let's let's go with you, man. Also, I have no idea how happy I am to see you again.

Speaker3 02:53

Then this probably won't surprise anybody. I'm kind of an antisocial person. You know, I like computers more than humans, generally speaking. So the good man, that's that's where I go. I really don't know of like any sort of like community that you're describing where it's not public, right? You can go to Stack Overflow and the associated websites and ask questions there. But I usually just Google and I find usually what I find, what I need. Generally speaking, so I'm a bad person to ask.

Harpreet 03:24

Yeah, well, that's the mark of a very expert Googler, because it's very hard sometimes for me to find the answers that I'm looking for. But yeah, like I was spending some time on on stack exchange. There's so many different parts to stack exchange, right? There's Stack Overflow. That's like the overarching sorry site, but there's one that was called like cross validated or data science. And there's like the Data sign stack exchange. And it was just it's crickets, man. People are asking questions

Speaker2 03:50

Regularly, but these questions

Harpreet 03:52

Go so unanswered. And I'm just like, Oh man, I wonder if that's

Speaker2 03:55

Is that

Harpreet 03:56

Someplace where I could step in, maybe help people and provide some value give advice. You know, I tend to like answering more technical questions and getting kind of tired of the how do I break into data science type of questions? That's another story for a different day. But Eric Eric Riddick, good to see you here, my friend. Nice setup, by the way. Where do you go, man? Where do you go for, um, for for information kind of kind of like, you know, to get help or else do you hang out on on the web when it comes to data science and machine learning stuff? By the way, if you guys do have questions, feel free to drop them in the chat or in the comment section on LinkedIn, on YouTube, wherever it is that you're watching.

Speaker4 04:37

I mean, since since you asked, I should say I

Speaker2 04:39

Haven't figured it out, but also guys,

Speaker3 04:41

Look, you like this microphone and arm? It's Harpreet Sahota. Yeah, I don't know. I ask around at work.

Speaker2 04:50

We have a

Speaker4 04:51

Band. There's actually some pretty legit, really friendly guys and women who who are pretty helpful.

Speaker3 04:58

But sometimes I just get stuck on like systems questions. And that's that's when I feel like I really don't know where to go. Like, for example, you do stuff with comet and like, like, you'd probably be the guy to ask about, like experiment tracking and managing like model metadata and stuff. But actually, I feel like that like that stuff about actually putting models in production still is something only like a very small sliver of like these people in my network are talking about and thinking about. And so I feel like when it comes time to build systems, which is my job, I don't really have a lot of resources, so.

Harpreet 05:31

And awesome, thanks for thanks for sharing that. I'm curious, like what what is like a typical systems type of question, if that makes if that question makes sense.

Speaker3 05:43

Yeah, good question. Yeah, like I could ask you guys, like, how do you like, what's your feature story? You know, like

Speaker2 05:50

How how are

Speaker3 05:51

You guys working on a feature story? Some people use ElasticSearch, you know, ask, which is nice because it has like a built in vector similarity search. And that's the way a lot of people do recommendations like recommender systems. So you could use a data lake or another question is like, how do you guys get your inferences available to the rest of your company?

Speaker2 06:09

Like, like you might

Speaker3 06:10

Develop models and make recommendations about something. But does that end up in other software like do you guys have like like an API that people did a gateway? I don't know. So things like that?

Harpreet 06:22

Awesome. Thank you. Yeah, thanks for providing that additional context. There is the trinity of Eric's in the building, so I hear we've got the three Eric's Eric Gitonga, Eric and Eric Sims, Eric Sims.

Speaker2 06:36

Where do you hang out for

Harpreet 06:38

For other questions besides like LinkedIn? And then let's go to Ken after that, Kenji, then then Vinicius. Then then I'm just waiting for questions to come in. So if you guys got questions, please do let me know. So go for it, Eric.

Speaker5 06:54

Yeah, so like Dave, I start with Google and then just go from there and I found myself recently, I found some really helpful answers on Quora because I was looking for some, like I was looking for recommender system project ideas, something that would help me connect with like an interesting Data or something, just that I hadn't considered before.

Speaker2 07:16

And it was a really

Speaker5 07:17

Smart guy there who, like, listed out some data sets I'd never heard of. And so it was just like a really helpful, well thought out answer

Speaker2 07:24

That I wouldn't ever really see a

Speaker5 07:26

Post on anything like that. So I don't go to Quora first, but I end up there once in a while. The other place that I sometimes go to is ResearchGate. When I Google Things and ResearchGate is like consistently way over my head. I wish, you know, one of these days I will have better math chops and I will get it. But usually I just get on there. It's like I came to the wrong party and I try to back out slowly. But it's definitely I mean, it's a wealth of knowledge as long as I can actually consume it. So there are probably a couple that come to mind.

Speaker2 07:59

Just curious about the recommender system, like, what's your

Harpreet 08:03

Project kind of looking like? What's your ideation for that project?

Speaker5 08:08

Yeah, so I would I found like from that from that project, I found like like like a dating website Data set. So I want to make a silly recommender engine for Valentine's Day. I don't know if I'll have something ready in time, but it was like, you know, a fun, like, collaborative filtering thing that I could use or practice with. But what I what I really want to do is create one that leverages diversity and serendipity to help us to help avoid polarization in like search search results, because that's what happens so readily now. And so how can we do something? The way I think of it is like, can we make a recommender system that makes this better instead of just making giving us more of what we want?

Harpreet 08:50

Yeah, no. You share that you shared a research paper along the the lines of that, that injecting serendipity into the recommendation systems. I thought that was super fascinating by any chance might if you want to collaborate on that. Hit me up, man. Let's let's let's do something. I think they'll be a lot of fun. Kenji, go for it.

Speaker4 09:14

So I have a lot of thoughts on this. This is something I am constantly asked and and I also think I think about. So the first, obviously for me is Google. Like everyone else, I

Speaker2 09:28

Google everything immediately.

Speaker3 09:30

But I've seen a lot of people

Speaker2 09:32

Have success with YouTube

Speaker4 09:34

As well. So YouTube is like a a little bit more intimate question asking platform. Not everyone's going to read it. It's not like a Reddit style thread where things get get. Super well-publicized,

Speaker3 09:49

If you have a personal

Speaker4 09:50

Question, you ask it and the the creator is engaged in their community, a lot of the times you'll get a pretty good and detailed answer there. Also Discord servers or Slack channels. Have been there More recently, I think, may be. And he's also it sounds like that community is really strong and a great place to ask relevant questions.

Speaker2 10:18

I also

Speaker4 10:20

Like Kaggle and GitHub,

Speaker2 10:22

So if you're

Speaker4 10:23

Having an issue with a certain library, there's literally a systems in place for you to ask questions about it. And there's also places where other people are having problems with that same library. You'll be able to to evaluate it and understand it better. You might find exactly what you're looking for, right? And then Kaggle, it's literally a place for a data scientist. They have a forum type architecture as well.

Speaker2 10:46

It's perfect for

Speaker4 10:48

For asking the right types of questions there. So I don't know if that's exactly exhaustive, but hopefully that's a little bit more outside of the box

Speaker2 10:56

And people might find

Speaker4 10:58

Something that fits their specific question style across one of those categories.

Harpreet 11:05

Awesome, Ken. Thank you so much. Yeah, shout out to Dimitrios with the ops community. That thing is this bump, and it's huge. Lot of stuff going on there. The thing with with like Slack Communities and Discord communities, at least for me, I I just feel like I can't keep up, man. Like it's, you know, it's extremely difficult to keep up on things. I guess that,

Speaker2 11:29

You know, we got that, that Charlie

Harpreet 11:31

Data thing, which I tried to be a part of, but then just got left in the dust because there's the movement so quick, right? But yeah, yeah, that's the challenging

Speaker2 11:39

Part to

Harpreet 11:40

Me, at least about Slack Communities, and Discord is just the pace at which it moves. I find myself being more of a fan of forums

Speaker2 11:49

Where the questions are more searchable and

Harpreet 11:54

Indexed a little bit better, and they they show up on search engines and stuff as well. Then what about you? And then after Venn will go to? We'll go to Eric's question about sequel, and then if anybody

Speaker2 12:05

Else watching on

Harpreet 12:06

Linkedin or here in the

Speaker2 12:07

Chat has a

Harpreet 12:08

Question, please let me know I'll add you to the queue. Also, just quick shout out to some of the people that joined

Speaker2 12:13

Since we started

Harpreet 12:15

Jennifer Nardin. What's going on? Marina is in the building to Becky Goes here as well. Kristen, haven't seen you in a long while, Kristen. I could see you and Matt Diamond then go for it.

Speaker4 12:26

I'm going to give the old man answer, because I think everybody's covered the really good communities as you go in your

Speaker2 12:32

Career and you meet people

Speaker4 12:33

That are way smarter than you. Stay in touch with them and start building communities like little private communities because the smartest people on Earth will spend, you know, a couple of minutes here and there answering questions. But I found and stumbled my way into these little groups

Speaker2 12:52

Where there's,

Speaker4 12:53

You know, it's between five and 10. Some of them are a little bigger than that. You'll meet up about once a month, but everybody is and a resource to everybody else. And the questions that get

Speaker2 13:04

Asked,

Speaker4 13:06

Like I've learned more from watching other people ask really smart questions and seeing the answers than I ever have from my own questions. I mean, it's a broader approach when you have a little, you know, a little group like that. And as long as everybody is pulling their own weight and able to contribute and help in one way shape or form answer questions those groups. It takes a while to build them up. But those groups are invaluable because everyone actually kind of jumps in and jumps on questions. It's almost like one of these, but you have access to it by email or text or slack or whatever. So build these up.

Speaker2 13:43

And if

Speaker4 13:45

You can curate one, if you can be like the person that

Speaker2 13:48

Brings everybody

Speaker4 13:49

Together and starts introducing new people to the group and everything like that, you become such an invaluable person because when you get to a certain level of question, it's like, where are you going to go to ask that? Mm hmm. You know, and so you have to have a different type of group if you want to get any sort of answer that's comprehensive and not, you know, oh, that's a stupid question. Or the the default five answers that you already knew, tried and didn't work. So build up these groups for yourselves. They're they're valuable and try to join one. You know, if you see people meeting on a regular basis, say, Hey, can I get it on that?

Harpreet 14:27

It's like we got going on here at the Data Science happy hours. I think that's one thing. I feel like that done a great job that was was kind of building this this space every Friday for us to get together. But I haven't had that success translate over

Speaker2 14:40

Into my

Harpreet 14:42

My own Slack channel for for the for the @ArtistsOfData science that it's crickets. I'm wondering what I could do to make that a better

Speaker2 14:50

Place, a place where

Harpreet 14:51

People can come with with more questions. You know, tons of people join every week, but nobody's I feel like utilizing that resource can go for it.

Speaker4 14:59

So that's something that I'm always thinking about is you create these communities and you sort of need a catalyst a lot of the time you need people that are going to be engaging consistently. That person can be yourself. But it might be some self-selection. But I think that empowering people who are active or early active to be more engaged, to be administrators or to to have more involvement and to think creatively

Speaker2 15:26

About

Speaker4 15:28

About how to improve

Speaker2 15:29

The quality

Speaker4 15:30

Of the forum is is awesome. I mean, that's something I've seen with the sixty six days of Data discord is that there are a couple of admin there that take it really seriously. They're super involved and I try my best to like, give them opportunities to put their own spin on things like a couple of them have made like bots for the server that like, removes spam. They do really incredible stuff that I have absolutely no clue how to do, and every one of those is incredibly cool, right? So the idea is that the more autonomy you give them, the more control, the more they feel like it's theirs, the more engagement they're going to. Drum up and other people are going to see that and want to be more involved as well. There's also some really cool ways on Discord to gamify it. So if people comment they they raise their level and things like that, which I think is so cool for some reason, I'm not even the highest strength in my own server, which is which is wild, right? But again, I don't know as much about the Slack

Speaker2 16:32

Platform, but that is

Speaker4 16:33

One of the reasons why something like a discord, which is very developer friendly. Could be a good option as well. Yeah.

Harpreet 16:43

I didn't even know you had any discord for sixty six days of Data can go ahead and drop a link to that so people can join that. I'll be happy to join that. I think like, like I mentioned, one thing I want to do is just, I feel like I got a lot of like,

Speaker2 16:54

You know, knowledge with

Harpreet 16:55

Machine learning and stats that I do not get to share as often as I would like to. So. I just need an outlet for that, need an outlet for that. But thank you so much. Shout out to Dylan in the house doing good to see you again. Eric, let's jump to your question and then anybody joining in on LinkedIn. If you guys got questions, drop your question right there in the chat. Or if you're watching, smash that like wherever you are.

Speaker5 17:19

So this afternoon, my boss messaged me and said that they'd like to update our interview process, and one of the things they want to include is a sequel assessment, like when I was interviewing for it, I had like a take home project, but I didn't have a sequel assessment as part of it. And so it's the setup for it. We have credibility. I guess as a platform, I've never heard of it, but it's just one of the, you know, you can live life code on it. The idea for the

Speaker2 17:49

Interview is that it would

Speaker5 17:51

Be a 30 minute call with the candidate, so I would actually be talking to them and

Speaker2 17:57

Then having them answer

Speaker5 17:59

Work through any any number of questions for roughly twenty five of those minutes and assessing whatever it is that we feel like should be assessed. And so I wanted to hear your thoughts on what would be what would you do if you were trying to put that together?

Harpreet 18:17

I want to go to Dave Lingam for this one. He's laughing, he knew I was going to come to him.

Speaker3 18:23

I'm not I'm not laughing because I thought you were going to me, I'm laughing because I hope you wouldn't. Oh, it's going to be hard for me not to go into a diatribe around the ridiculousness of interview processes these days. So I'm going to try and keep that on the side, Eric, as much as I can. So first up, what kind of role are you talking about? Because the kind of sequel that you want to test is going to be highly determined on the role.

Speaker2 18:52

So, for example.

Speaker5 18:53

Great question.

Speaker3 18:54

Don't ask Data analysts about indexing. I mean, come on, really, that's a dba kind of job for first and foremost, right? Maybe ask them about how they might use an index to make the query faster, but not like it's a clustered index or something like that. It just is making any sense. So that would be the first thing I would do is like, what's the role? So fill me in on that.

Speaker5 19:14

My job, essentially. So, analysts, analysts, stuff, not DVA. I don't even know closer to indexing is so not going there.

Speaker3 19:21

Ok, sweet. See, you prove my point. You prove my point, right? I would, you know, if you guys are familiar with me, you know, I'm a big fan of of principle. 80 20 rule, right? There is a 20 percent selection of SQL Data analysts and Data focus. People use about 80 percent of the time. So it's going to be the normal kind of stuff, right? Can you do? Can you use window functions? Can you use case when within, you know, inside of Group AIs and that sort of thing to actually create interesting transformations of the raw tabular data into something that's a little bit more aggregated and useful for data analytics. That's that's typically where I would start. In terms of like the surface area of kinds of SQL, you would do, and maybe it's as simple as, I don't know this platform that you're talking about, maybe it's as simple as having a couple of different tables that you show them through screen

Speaker2 20:15

Share and then say, OK,

Speaker3 20:17

Join these up, and I want you to be able to create like date based count features or very, very good. That's a very, very good

Speaker2 20:25

Indication that somebody in my

Speaker3 20:27

Experience has enough SQL knowledge to actually do, you know, has enough SQL knowledge in the data analytic space. So that's kind of what I would do. I would start with simple tables, ask them like an 80 to 20 kind of kind of question and see if they can work their way through it.

Speaker5 20:43

Oh, maybe if if you don't if anybody doesn't have something else to add or if you do great. But if you have examples of good or horrible assessments, I mean, I have had one online skill assessment and it was terrible. The thing that was most stressful to me about it was that it was not live. I didn't have a person that I could talk to. It was just like, you either get this or you don't get this and you're on your own. It just didn't. I didn't like that feeling. But you know, if I'm going to be one administering it, I can. I take it on myself to be a human being, talking to a human being and try and help set them at ease because that's what I can do. So, yeah, good or bad examples are also appreciated.

Speaker3 21:27

Yeah. So I resonated with Vince comment earlier about being the old man. I think I'm probably the oldest person on the call,

Speaker2 21:35

So I'm going to give you

Speaker3 21:36

An old man answer perspective. Try to simulate as much as you can a whiteboard interview experience using a virtual technology. That would be my advice, which is, of course, as you indicated, would be live, right? Show them a table on a screen, walk them through it. They're going to get stuck. That's OK. I don't expect people myself to. I don't expect them to know the syntax like backwards and forwards. I mean, this idea that somebody has spent four hundred hours on leap code studying SQL for one interview is just insane. It doesn't

Speaker2 22:08

Matter in the end, because once they

Speaker3 22:10

Land the job, they're going to Google the syntax anyway.

Speaker2 22:11

It doesn't matter. So what's more

Speaker3 22:13

Important is understanding what is their basic level of understanding of the core concepts, and if they screw up a little bit of the syntax, who cares, right? But you can only evaluate that live.

Harpreet 22:24

Is is kind of the it's I don't know if this is a good way of thinking about the question.

Speaker2 22:30

I guess my question

Harpreet 22:31

Is, is this a good way of thinking about it? Is like, what's like the most reasonably challenging question you could ask that would be indicative of other skills or indicative of, like, does that kind of make sense like indicative of

Speaker2 22:48

Lower level base level knowledge, I guess.

Speaker5 22:53

Acts like if one thought that I was wondering was like like speeches, for example, like, do I really have time or like the complexity to set something up where it'd be reasonable for someone to need to put something together like that? It's something useful. It's something that I use frequently. But like, I don't really know that there's necessarily in 25 minutes. I don't really know that I got time for something like that. So how do I assess it otherwise, you know, through conversation or whatever?

Speaker3 23:23

The example that I gave

Speaker2 23:24

And I've interviewed a lot of

Speaker3 23:26

People in my time, that's why there's all this gray in my beard. For SQL count count based features using like using a date column is like the one of the single best things I've ever found to actually ascertain someone's general level of skill knowledge for pulling data out of a relational database and transforming it into something that's useful for analytics. Because you've got to hit, you've got to do a group by, you need a case when you need to work with Daytimes, all at the same time. So it's a reasonable like it's a

Speaker2 23:57

Reasonable one problem kind of thing that assesses

Speaker3 24:00

A lot of people's knowledge regarding how they use

Speaker2 24:02

Sequel with a relational

Speaker3 24:04

Database and then transform that into a feature set that's useful for analysis.

Harpreet 24:10

Then what are your thoughts here?

Speaker4 24:14

But in the comments, and I know this is going to sound sarcastic. But one thing that I would actually do is like, give a fairly hard. You know, I said ridiculously hard, but fairly hard question. And then say, Hey, Google it, Google. The answer, unless you know it off the top of your head. And based on the Google that you just did, you come up with a actually implement what you googled? Because that's kind of realistic. I mean, we run into stuff we don't understand all the time or just forgot. And if we could add that to interviews and actually included in the process, we're like, Here's a question I know you don't know. But the whole point of this is to figure out how fast you can figure it out. And can you then once you figured it it out, implement, because that's one of our, you know, I think across the board is data scientists. It's one of our biggest skills because there's so much that you can run into

Speaker2 25:08

That there's no

Speaker4 25:08

Way you've ever run into before or there's complexities that you run into and you'll ask the team and they'll go, I don't know, Google. So it's I don't I feel like it's more realistic.

Harpreet 25:21

The single most important

Speaker3 25:22

Feeling how we spent.

Speaker2 25:24

Oh, yeah, go for it. Go ahead.

Harpreet 25:26

I was going to say single most important skill in data science is resourcefulness, and that would be one hell of a way to test for resourcefulness.

Speaker5 25:33

Oh, certainly the first 20 minutes of this conversation talking about various variations of Google. Yeah, that's definitely fitting.

Harpreet 25:42

Russell, you had some thoughts here as well. As I scroll up through the chat.

Speaker2 25:51

Yeah.

Speaker6 25:51

Hello, everyone. I started in a couple of comments in the in the chat window. I'll start in reverse order, so I've just responded to VINs comment about Google, and I was saying very often I'll still Google about simple questions, simple solutions, just so that I see what's out

Speaker2 26:11

There in

Speaker6 26:13

The in the database, basically see if there's better ways to do things. I mentioned anger bias or anchoring bias, you know, so that I haven't chosen my favorite solution that I stick to and keep doing the same old thing for five years. Even though a better procedure may have been discovered, you know, three years ago, you know a more simple way. So I like to try to make sure that I'm implementing best practice from a, you know, a respected peer forum, but also to see what else is out there, good and bad. So if there's bad methods out there, I like to have visibility of them so that if I see anybody else doing them, I can say, Well, yeah, I can see that that's popular. But you know, that's not the best way. Try looking at this way, you know?

Speaker2 27:00

So Google is

Speaker6 27:05

It's a it's a great resource, but it is volatile. So if anybody looks at Google for anything, you know, look at the 10 things, 20 different responses. Don't look at the first thing that comes back at the top of the page. I mean, very often those are ad generated anyway to try and look past the ads, but even then go down, you know, to the bottom of the page go two or three pages in trying take like, you know, an average of the answers that come through the ones that are getting good feedback. And are there a lot? Those are the ones to trust rather than the the odd one that's in there that, you know, may look good on the surface, but if there's no validation to it and you try to implement that, you may end up in a pickle.

Harpreet 27:48

Russell, thank you very much. Shout out to Greg Kilkeel in the building that you, Greg. All right. So question coming in from LinkedIn. I think he's here above him.

Speaker7 28:01

So I'm looking for some help and guidance around how the organizations are establishing some

Speaker2 28:09

Ethical

Speaker7 28:10

Practices. We're working on some things,

Speaker2 28:12

But this is

Speaker7 28:14

A great forum to get some inputs from everybody

Speaker3 28:18

Here. So thank you so much for any of the guidance that you're providing. I had that one question.

Harpreet 28:26

So definitely. Great question. I just want to say I don't have an answer off the top my head, but I just want to let you know that on February

Speaker2 28:34

5th, I believe I'm doing a

Harpreet 28:37

Podcast interview with Grant Fleming, who is the author of a book called Responsible Data

Speaker2 28:43

Science. I've yet to go

Harpreet 28:44

Through the book where I've gone through a little bit, but that might be a book worth checking out. I did skim

Speaker2 28:49

Through it, and it seemed very practical,

Harpreet 28:52

Practical and very applicable. You know, there's these apply those concepts, so definitely check that out. But in the meantime, does anybody want to jump in here, provide some insights here for and just he's looking for trends and organizations for establishing ethical practices

Speaker2 29:10

And maybe some good resources for guidance. I wonder if

Harpreet 29:15

Is Makiko here? I feel like we would have good response.

Speaker8 29:19

Hey, where are you? Oh, here you are.

Harpreet 29:24

Yeah, I see you now. Yeah. Any tips?

Speaker8 29:31

Yeah, I think it's kind of interesting because it's like ethically AIs the thing that everyone wants to do, but not everyone's frankly capable of doing it

Speaker2 29:39

Because

Speaker8 29:41

A couple of things, right, like one, it requires like monitoring and observability sophistication or maturity, which not a lot companies do have that. Yes. Secondly, and there's a couple of other things to write like, so that's like mall performance, right? Then the second part that you sort of need is you kind of need AIs on the ground when it comes to the Data.

Speaker2 30:06

So you could argue

Speaker8 30:08

That then that starts touching like data governance, provenance, lineage

Speaker2 30:13

And

Speaker8 30:14

Then at the other day, like you need to be understanding how to connect like the data science and machine learning sort of efforts

Speaker2 30:25

With like

Speaker8 30:26

The broader business goals. So, for example, way back one, my favorite example in the world. So there was some

Speaker2 30:33

Residential,

Speaker8 30:34

Some real estate tech company, which may or may not have been

Speaker2 30:36

Zillow. I also don't

Speaker8 30:37

Remember it could have been Redfin or someone they were trying to roll out this idea to essentially do, like white glove treatment for people who were listing properties above a certain value. Now the problem with that is that property value is may or may not be correlated with things, for example, like race, like your economic, socioeconomic status,

Speaker2 31:04

All that

Speaker8 31:05

Stuff. Because in the U.S., we had this policy called redlining, where essentially, you know, people of

Speaker2 31:12

Color were not.

Speaker8 31:14

They're not provided the same interest rates and loans to purchase property.

Speaker2 31:19

Right, so,

Speaker8 31:21

Yeah, so there's like a lot of huge issues

Speaker2 31:23

There.

Speaker8 31:24

But I would say, like the first couple, like the three major components I see, at least on the technology side is one having good observability and monitoring is like absolutely key, both on like machine learning or both on your models and your data.

Speaker2 31:39

The second part is.

Speaker8 31:43

It's definitely like having AIs on the Data because of the distribution

Speaker2 31:46

Changes, like irrespective

Speaker8 31:47

Of the models, you kind of need to understand that ahead of

Speaker2 31:50

Time. Third is having a

Speaker8 31:53

Good connection with legal, to be honest and law of companies. I feel like sometimes that relationship isn't there until like something bad happens. I wish the eagles them like, Oh, why did you like hopscotch around us? And then the Data science leadership is like, Well, because it takes forever to get through you guys. It takes forever for you all to do like review. And I would say, like the start of the fourth aspect is like. Having a conversation on. Like, how do the broader gate and machine learning efforts fit into the company in the business and the product? And like how that is connected with like the users? So a lot of words to say that like there are people who are doing ethically who are involved in that, Facebook has a fair Microsoft, Amazon, they all have groups that are focused on that. But when you start getting away from the big tech companies or like the companies with the huge resources, it becomes like. Frankly, more of a talking point as opposed to like something that people are like actively involved in. And if they are, it's like a side effect of them investing in like their monitoring and production efforts or they're monitoring

Speaker2 33:06

Observability, tracing and

Speaker8 33:07

Like data governance efforts. So sorry,

Speaker2 33:12

I should've been more useful.

Speaker8 33:13

But yeah, you know, if you're getting started like the Rio, like, you know, not everyone's there yet.

Speaker2 33:20

Yeah.

Harpreet 33:20

And there's tons of resources in the forms of links to like white papers and stuff in the chat. And as always, all the links will be in the show

Speaker2 33:28

Notes for this

Harpreet 33:30

Javier session as well. So feel free to click through on your leisure. Anybody else has any input here? I'm curious. So, you know Ben or

Speaker2 33:39

Dave or

Harpreet 33:41

I got a question like, what's the what's the connection between like data literacy

Speaker2 33:47

And like

Harpreet 33:49

Ethical data science if if there is one day? What are your thoughts on that?

Speaker2 33:54

Yeah.

Speaker3 33:55

So I guess the question then becomes is like a base. What is data science? First and foremost, right? And I know some of you, if not most, you're going to roll your eyes going, Oh God, we got to go in this again. Because for a lot of folks to social media, for example,

Speaker2 34:11

Data science

Speaker3 34:12

Equals equal machine learning. So if you if that's what you feel, then then there isn't really a ton of overlap, actually.

Speaker2 34:20

If you think

Speaker3 34:20

Of data science essentially is this kind of umbrella term for being able to do analytics right, use data to make a more effective business in some

Speaker2 34:30

Way, then the relationship

Speaker3 34:32

Is one of a maybe of a maturity spectrum or a a path of

Speaker2 34:37

Development where you start

Speaker3 34:39

With data literacy, where you start with the basics, right? Where what is Data? How do I understand data? What is how do I understand the data is collected? What usefulness is this data provide me? How do then do I use the data in a relatively standardized way to achieve some sort of outcome? For example, how do I make better decisions? How do I analyze what's going on with my KPIs, that sort of thing?

Speaker2 35:01

And then along the

Speaker3 35:02

Spectrum, right along that journey, you move more into the data science kind of space. So I would say the related, depending on how you define data science. And these days, I'm not quite sure what the popular definition is, is because I don't pay any attention to it anymore. What free?

Harpreet 35:20

Let's go to let's go to Vin, and then after Vin Russell Russell's got some great comments here, and if anybody else has any input here, feel free to just raise your hand icon. I just think you talking

Speaker2 35:31

About ethical trends in

Harpreet 35:35

Establishing ethical practices

Speaker2 35:37

And perhaps

Harpreet 35:38

Do some good resources for guidance.

Speaker4 35:43

Yeah, I think the first thing to note is I'm going to sound really optimistic after saying this, but I have to lead off by

Speaker2 35:48

Saying every business

Speaker4 35:50

Is only as ethical as their

Speaker2 35:52

Bank account and their customers

Speaker4 35:53

Allow them to be.

Speaker2 35:54

So, you know, when you

Speaker4 35:57

Build out this at the very beginning, it's easy to think, Oh, I've got a mandate, they're always going to stamp it. No, they're not. Now that being said, most companies are actually worried in some way shape or form about ethics when it comes to AI because they're worried about getting sued. They're also seeing regulations come down U.S., Canada, Europe,

Speaker2 36:18

They're

Speaker4 36:18

All. I mean, China's got

Speaker2 36:20

A robust

Speaker4 36:21

Set of regulations as well. I mean, they're they're coming out of a number of different countries. So the fear is we're going to get sued, we're going to have a massive fine. And for companies like Google and Facebook, they legitimately do not care. And that's the that's the barrier that you're always going to run into when you're building an ethics program

Speaker2 36:39

Is at some

Speaker4 36:40

Point, someone is going to question, well, if no one else is paying attention to these rules, why should we? And so when you start one of these campaigns, when you start one of these programs, you have to start with

Speaker2 36:52

A

Speaker4 36:52

Doctrine and it has to be adopted by the entire company. And that's really an and it's an honest doctrine. You can't you can't be aspirational when you build this. You have to go through and talk to individual business units and senior leadership and say, OK, what do we really, really most concerned with? What are our actual principles? What are the things that we're worried about? And those are going to be part of that doctrine? And everything makes more sense when you have some basic founding principles and some guidelines where you say we won't do this, we won't do that. We will go to this level of effort to make sure that we don't have these negative impacts. Here's what we define as negative and bad. Here's what we define is good, because sometimes what one side thinks is a bad impact. Data science and machine learning you as a company might not agree. And so build that out. It's not. It doesn't have to be a huge document. You're not writing a religious thesis.

Speaker2 37:57

You really

Speaker4 37:57

Just coming up with the core principles for everything that you do going

Speaker2 38:01

Forward and you can have everyone

Speaker4 38:03

Sign that. So everyone throughout the organization who is talking about projects, who's building projects has now signed on to an ethical standard, and that's something you can hold them accountable for. You can put that in senior leadership skills. You can, you know, and this is why I say it has to be it has to be realistic or none of that other stuff happens or this document gets emailed and goes into the ether and someone's archives somewhere where no one can find it. So really, you know, I can we can talk about ethical frameworks because there's tons of them. But I think if you don't start with that foundation of just explaining what the company really cares about and what the company is actually willing to do and what the company will not do, you don't have any sort of success criteria going forward, and you can't really you can't build rules for every scenario that is ever going to come up and say, if you have a set of guidelines, at least people can look at that guideline and say, OK, I tried to stay in, you know, in the spirit of this guideline, or I have no idea, I don't think this covers anything. I need to go talk to somebody.

And half of it is that second scenario where you run into something and you

Speaker2 39:12

Go, OK, I

Speaker4 39:13

Remember signing this document, and I don't know. You know, it's just that awareness piece that sometimes is enough to surface ethical concerns, and it gives a place to go someone that you can talk to if that ethical concern comes up. And so with that foundation, I think you'll be way better off no matter what direction you go in,

Speaker2 39:33

But also

Speaker4 39:34

Don't get shot. Don't don't let this be the thing that gets you fired because you are more passionate about it than the business is. You know, I'll conclude with that. Most businesses have great, great intentions, but at the end of the day, the bottom line and the shareholders are the bottom line of the shareholders.

Speaker2 39:50

Ben, thank you very much.

Harpreet 39:51

Russell, you got some great comments. Anybody else wants to provide

Speaker2 39:54

Some, you know,

Harpreet 39:56

Any any insight on this. Please do libido. Well, I just think you or if you got questions on LinkedIn or questions here in the chat that you want to ask, please go ahead and let me know and I'll put you into the queue.

Speaker6 40:09

I was just building upon his earlier comments about, you know, bank plan analysis and bias and prejudice and that.

Speaker2 40:19

And I was simply

Speaker6 40:21

Identifying that the Data presence in that industry prior to the advent of AI could easily have allowed for better analysis and better decision making. And the reason it didn't is because the people, as Vin was mentioning, who held the purse strings and maybe the entire organization,

Speaker2 40:43

If there

Speaker6 40:43

Was institutional opinion and bias, chose not to. So just as important as the AI or the ML models themselves in making decisions is the awareness and acknowledgment of

Speaker2 41:02

Cognitive biases in

Speaker6 41:05

All decisions. So that's decisions of interpreting their Data decisions of building the model in the first place. You know, if you selectively choose a spread of a thousand, say, photographs, for example, to do some image recognition and you bias that selection so that there's 90 percent of the thing that you wanted to discover and 10 percent of everything else. And your model is going to give you the answer you want, but it's not going to be the right answer. You know, and the same thing applies to everything. So I was just really drawing attention

Speaker2 41:37

To the fact that

Speaker6 41:38

In the last, say, five to 10 years, I've seen a lot of, let's call it noise or a lot of reporting about cognitive biases. And you know, there's 40 or 50 there if you really want to drill down into them. But some of the most common like anchoring bias are confirmation bias, etc., etc. So. Having the people that are in charge of building the models in the first place, make sure that they are well versed in the biases and are aware of identifying early signs of bias so that they can try to direct the model in the right direction before it's built wrong. And you have to go back and fix it. You know, if it gets too far, it's going to be written off. You know, it's more economical to build a whole new model than go back and try and fix a model that's biased from the ground up.

Speaker2 42:30

And also the

Speaker6 42:32

People that digest and consume the output from those models try to make sure that they are educated enough to detect the first instances of bias that may not so much organically present itself, but present itself where there was no sign of it in the first instance. Yeah, I think that's it. So just be really aware of those biases.

Harpreet 42:57

Thank you very much, Russell. Some comments coming in, Kanji says, I think having a person outside of the team who has no stake in the project observe for morality and ethics as well, and also having non Data scientists evaluate for ethics is valuable.

Speaker2 43:13

Makiko said I don't necessarily

Harpreet 43:15

Think machine learning is a great tool for identifying bias personally unless someone is willing to do model in future analysis. Yes, interesting point. There's nothing inherently biased about, you know, a random forest or a convolutional neural network or any of those model architectures themselves. Nothing inherently biased about those. It's the data that you train on. So just be responsible with the features that you select and make sure that whatever samples you have, whatever data you have, is it's representative of. Everyone or everything, you know, I mean, just be as representative as possible.

Speaker2 43:52

Greg, see your hands up.

Harpreet 43:54

Go for it.

Speaker2 43:57

Twenty to

Speaker7 43:58

Go. I think a lot of great points were made when it comes to

Speaker2 44:05

Ethics, and you touched on

Speaker7 44:06

This, that the tool itself and by two, I mean the machine learning models, they're not unethical. So the best way to operationalize ethics is to embed it into your processes, build that framework first, defining what's ethical and what's not. And with that, embed this framework into your project lifecycle management. So if you think about the Data science project lifecycle management where it starts away in for each of these stages, you have a set of questions that you ask and see if they meet or don't meet the threshold for ethics versus not ethical. Then once you you do that you deploy, you have to have a mechanism for

Speaker2 44:55

What you call auditing.

Speaker7 44:57

So how do you measure that? How do you check from time to time when these models are making decisions that they're not drifting, they're not going towards the unethical barriers? So, you know, just this is where, you know, you kind of you have to bring this. What do you say, auditing framework to make sure you surface these areas? So lastly, just in case you missed it, I put on the side here that I do have a copy of a great dog from integrity.

Speaker2 45:39

They have something about

Speaker7 45:41

Responsible A.I. in consumer enterprise. They have a framework that show you how to operationalize this, and I believe you can abstract

Speaker2 45:52

It to

Speaker7 45:54

Other industries as well. So if you're interested in it, then it get your name on time. The best thing to do is to be on LinkedIn and I'll send you a copy directly there. It's probably better to do it directly on LinkedIn versus send you an email, so I'd rather do that. So let me know if you're interested to read this Doc..

Harpreet 46:14

I think so much Russell is asking to be possible to create an independent model solely for the purpose of analyzing the output of another model to identify evidence of bias. Yeah. These generative adversarial networks. For that, there is a link to a paper that will be in the show notes so they can

Speaker2 46:34

Check that out. It's called generative

Harpreet 46:35

Adversarial networks for mitigating biases and machine learning systems. So, yeah, definitely check that out, Makiko. Go for it.

Speaker8 46:43

And I think these three random points, like one like a was so something that helps us don't have racist people in charge of policy that that helps a lot, right? Because with redlining. Right. And what was the early iteration of Freddie Mae and

Speaker2 47:05

Freddie Mac,

Speaker8 47:07

Fannie Mae,

Speaker2 47:08

Right?

Speaker8 47:09

They knew exactly what they were doing, right? Like, that's where the term came from was. They had maps where they literally

Speaker2 47:15

Said, OK, this one, this neighborhood,

Speaker8 47:17

Let's color and red, and this one is yellow and green and somehow corresponds to ethnic groups and all that stuff, right? So they kind of knew exactly what they were doing. So if you think about it, the system worked great for them. It worked terrible for like the end users and the people who are impacted by that right, because they've done studies to show that a lot of the generational wealth that people of color could have had in the U.S., a lot of that can. You can tie it back to essentially the kinds of interest rates and the kinds of homes and property they were able to purchase during that time.

Speaker2 47:53

Right.

Speaker8 47:55

So, you know, so I think that you can have a perfectly working system, but that is still kind of bad. So that's that's one thing to point out, right? So we do have to hold our leadership or people who are in charge of all that stuff accountable. I think the second part to right is I think the term bias is a little bit overloaded. So I was talking with Serge Macias about

Speaker2 48:17

This, but Yann

Speaker8 48:19

Lecun, he gone to.

Speaker2 48:21

There's a little bit of a kerfuffle going on

Speaker8 48:23

At a while ago because he had said there's no racist models. And. But that's not to say that machine learning cannot be racist, right? What he is saying is that like it really is the Data and essentially like how you structure the model and like that will kind of impact the results you get and also the different segments of people that are impacted. And I preferred that explanation because when we say the model is racist, we're like essentially deferring accountability and ownership for making the system, for making the pipelines, for making our products better onto the tooling, as opposed to the people who design and manage the tooling and the experiments, right? So I think that's something to consider is that like it is a little bit of neural overload a term. But on the other hand, if you have all the right

Speaker2 49:13

Data models like machine

Speaker8 49:15

Learning models, it's like all things right. Like if you only predict off the Data,

Speaker2 49:19

You give them right, the only

Speaker8 49:21

Predict off the features that you give them. So something

Speaker2 49:24

That you can

Speaker8 49:25

Do right, for example, is you hypothetically, if you had like protected data about people you could like, like analyze them all the outputs and say, like, OK, well, this is saying, let's give a lower interest rate to people of color than non people of color, right? So but at the same time, it's like, should you be analyzing off? So it's one of these things where is like, it's such a complicated question, but I think there's a couple of things, right? Like number one, if it's if it's a question of like, who should be doing better, we should always kind of like, look to ourselves as professionals and as like teams to go, what can we kind of do to contribute to that effort? Maybe it's better observability. Maybe it's being more, you know, friendly to legal, getting them in the conversations more or something like that, you know, but also to like to a certain extent, we benefit off of models being very biased, right? Like we want them to give highly targeted predictions or what have you. So I do think it comes down to like companies need to. There needs to be incentives for companies to be willing to make a change. The incentive is not there. It's not going to happen.

Speaker7 50:35

So, you know, let's not forget the the complexity of it too. When you when you have a company that is transnational, right? So the values that you hold due to yourself in, say, say, is the same in China, right? Do they value dogs the same way we value dogs, for example, you're training an automated vehicle to roam around in the street. Is that vehicle going to do everything it can to avoid killing a dog here versus over there? So how do you build those ethical

Speaker2 51:12

Policies that these

Speaker7 51:13

Machine learning will be governed

Speaker2 51:15

Under to make sure that

Speaker7 51:17

You respect them and you don't get

Speaker2 51:19

Penalized?

Speaker7 51:20

So I know for sure it's almost impossible to remove bias from Data. Or at least you'll be running a rat race, but I'm also pro independently audited results. So if you think about underrepresented groups that are constantly being penalized by the machine. So when you pull the historical results of that machine, you can kind of

Speaker2 51:47

See trends in.

Speaker7 51:48

Those analysis can either be done by a machine or people to make sure

Speaker2 51:52

That be surfaced, those

Speaker7 51:53

Discrepancies so. Very good point here in Mexico. Thank you.

Speaker8 51:57

And actually another common another problem that we're kind of seeing pop up, for example, like in like the NLP space for people to chew over is pronouns. That is a big, big thing, right? So, you know, at MailChimp, I'm not going to talk about the work that we're doing right, but or in other companies where we deal with a lot of like text Data, right? Whether it's emails, whether you're a residential listing, posting or what have you you want to like, we want to make sure that, for example, if you're if you're trying to target to persons like business title or like. Marital stress or what have you? There's all this data that's available now that you can buy. I mean,

Speaker2 52:42

For years that you could buy it, enrich

Speaker8 52:43

Richard

Speaker2 52:44

Data.

Speaker8 52:45

But like, how do you respect like the pronouns? Do you? Do you even care about like. Like, bring that your analysis, should you bring that into your analysis, so that's like another thing that's really interesting to bring up because in certain like communities, they're like, why are we even caring? Like, why do we care about pronouns? And in others, it's like, Yeah, we should totally care, right? If we're going to care about, you know? Like race or culture or language or nationality, like we should care about pronouns, you know, and that I think comes down to the company, right, like some companies are, you know, very much like. You know, we we need to support policies to allow people to bring their whole selves to work and feel safe and all that jazz. And like you know, some companies like Coinbase, they're like, We're not going to be engaging in any progressive, I don't say progressive policies. You know, they basically said they're not going to be doing these efforts.

Speaker2 53:41

I think other stuff, right?

Speaker8 53:43

And they basically say, like, it's just not part of our mission. We don't care about. It's not like. So what is machine learning look like at Coinbase versus like what is

Speaker2 53:52

Machine learning look

Speaker8 53:53

Like at a company where they're like, No, we that's important to us like pronouns like making sure you know your nationality where you live, like, you know, all other stuff. So that's a part that's like it's not a technical problem, but it's like an internal conversation that reflects the business's priorities and all that stuff.

Speaker7 54:16

It's interesting, even if you. Oh, sorry about that. But. No, no. Go for it. I was going to say to the point of embedding that framework into the stages of that lifecycle, you know, for example, when you get to feature engineering, for example, you have to have the right questions, right? So if you're trying to guess. Uh, explore or classify or forecast whatever it is. Salary levels for people, you have to ask yourself, what does race have to do with it? Do you have to use that as a feature in your model, right? Is it really doing something right? So if if there's no mean, you have to bring up these questions to kind of ask yourself by ingesting this, does it really affect the result that you're trying to obtain? And sometimes you'll discover that a lot of these columns can be automated to prevent

Speaker2 55:20

Some sort of unnecessary

Speaker7 55:22

Bias to be ingested

Speaker2 55:23

Into into the

Speaker7 55:24

Models. So asking the right questions, asking the ethical questions is the way to go.

Harpreet 55:30

Yeah, just because the feature is highly predictive doesn't mean that you should use it to make a prediction. That's an interesting point, because think about the pronouns. I'm just wondering like if you're training on on Data for NLP tasks, I'm sure there has to be a cutoff point temporarily for the Data that you use because this modern usage of pronouns probably wasn't. You know. As pronounced No. Five to seven years ago. You know what I mean. But before we get to Vince's point here, there's a somebody on on LinkedIn. Brian had suggested Reed Blackman to follow Reed Blackman, CEO of Virtue and ethics adviser on LinkedIn. I post a lot of great content on a on ethical Data Science Article III. Before we get to then I just quickly want to read off this post. That's very relevant. His most recent post said that one of the most dangerous ideas in AI ethics community is that the Data scientists and engineers can by

Speaker2 56:29

Themselves save us

Harpreet 56:30

From the ethical risks of

Speaker2 56:31

Ai. Take each

Harpreet 56:33

Of the big

Speaker2 56:34

Three ethical

Harpreet 56:35

Risks of AI bias, black boxes and privacy violations. In each case, a large segment

Speaker2 56:42

Of the AI

Harpreet 56:42

Community thinks we can already solve this with technical fixes. We get, for example, mathematical definitions or metrics or fairness, lime and shap for explainability and techniques like differential privacy. But those metrics are incompatible with each other. Those explanations are unintelligible to the people that can be wronged by

Speaker2 57:03

Ai, and anonymous

Harpreet 57:06

Doesn't entail that people have control over their personal data. If senior leaders don't understand that these technical approaches to ethical risks are far from

Speaker2 57:15

Sufficient, we are in

Harpreet 57:17

Big trouble.

Speaker2 57:19

And then then in the

Harpreet 57:20

Comment section of that same

Speaker2 57:21

Post, which I'll share

Harpreet 57:23

Here in the links says that I doesn't give rise to novel ethical risks. It gives rise to novel sources of ethical risks with which you are all too familiar.

Speaker4 57:35

Vancouver Actually talking to Greg's point, the when you get into the the layers of different ethical frameworks with which you're actually getting into is different

Speaker2 57:48

Ontologies where

Speaker4 57:49

People classify things differently. Categorize things and their relationships between categories change.

Speaker2 57:56

And this is one of

Speaker4 57:57

Those big pieces where just awareness of ontologies and beginning to build out a value

Speaker2 58:04

Connected ontology,

Speaker4 58:06

It brings awareness to how someone could now use this. Every time I talk about this, I use vegetarians, vegans and carnivores.

Speaker2 58:15

When you start talking

Speaker4 58:16

About, you know, serving somebody in AD, do I serve this person and add for a steakhouse if I do that to someone who eats meat? Awesome. You know, I have connected with their ontology. My ad matches with their ontology. Someone who's a vegetarian, that's a different ontology. They look at meat and meat is not connected with food. Whereas for someone who's a carnivore, meat is food. And so when you start creating these ontologies, you inadvertently start gaining insights into some of the value systems. And so implementing ontologies is one way that you can start to at least understand some of these areas. And I'm not going into the really complex, controversial ones. But, you know, I think the food and that one's about as easy as it gets on. I'm talking about this topic, but you begin to see more complex relationships and value

Speaker2 59:14

Systems as part

Speaker4 59:16

Of your metadata around your Data catalogs, and those eventually build out ontologies. The hard thing is when you have conflicting ontologies where the majority of people overwhelming majority of people view things one way and there is a negative impact

Speaker2 59:36

Towards a

Speaker4 59:37

Very small group. And that small group,

Speaker2 59:41

Just

Speaker4 59:42

Overwhelmingly throughout history, has always been trampled because in some cases you don't even know they're there. It's the last thing I'll say is there was an entire group of autonomous driving engineers working on machine learning who learned by tweet that their computer vision datasets were missing trans individuals. And so there had been a couple of experiments done that showed that cars were more likely to run over people who were trans. You know, and this is one of those if you don't have somebody like that on your team, if you don't have somebody that falls into one of these groups on your team or who you want to say like that, I mean, who thinks that way

Speaker2 1:00:24

Because

Speaker4 1:00:25

You really do need someone

Speaker2 1:00:26

Like that who just thinks

Speaker4 1:00:28

Of every possible group, no matter how small

Speaker2 1:00:32

And has a moment

Speaker4 1:00:33

Of, well, we check what would we test

Speaker2 1:00:37

Because

Speaker4 1:00:38

There's just so many of these and you can have a crazy impact that just shows up out of nowhere because

Speaker2 1:00:46

They're thinking about it, just

Speaker4 1:00:48

Lack of awareness.

Harpreet 1:00:52

And thank you very much. Any other. Insight here, any other points? Got a question actually coming in from coast up, guys? Happy to help you stay on this point where we can move on to the next question. This one is not as touching a subject, but Kosta wants to know. Data science or machine learning teams in large software products, how do you guys go about task estimation?

Speaker2 1:01:22

Story points,

Harpreet 1:01:24

What's different to you and what's the same? How do you factor in experimentation? That's a good one, actually. Metro's test points pace. That's something that piece had copy shared something. Whatever match ups the other day about data science and scrum will go ahead, and I'll post that link here. I mean, there's some parts of data science like experimentation, for example, that it's kind of open ended. So I feel like just time box can get and then giving that thing like five points because it takes a lot of work. But but I'm curious, I'd love to go to to to Makiko, if you're available, if not, maybe a. Vin or Dave on this?

Speaker8 1:02:10

Yeah, it's funny, I think we're kind of struggling with this right now or we're we're dealing with it. I feel like it goes to the

Speaker2 1:02:17

Question of

Speaker8 1:02:18

Like, what do you do with like one off projects like is this like a recurring cast versus like a build

Speaker2 1:02:25

Project?

Speaker8 1:02:26

I feel like build projects you'd almost want to look at existing like software like frameworks for estimating storage points for regular software. But if it's something like, for example, you're consistently like if you're building a pipeline off of existing infrastructure, I would just look at how long it's taken before. Um, honestly, if it's like something that's like totally like new and it's I was going to say balls to the walls, but I got to figure out, I got to figure out more PC appropriate for this use, but I kind of almost feel like

Speaker2 1:03:01

It's you give the maximum

Speaker8 1:03:02

Number of story points. Make sure your leadership is like super nice. And you also caveat a lot of stuff like make it seem like it's going to take 2x longer than what you think it's going to take. So even if it only takes one point seventy five x longer, then it's like it's still within the bound. And then just try to scope down from there. I feel like what was it? The mythical, the mythical man month like? Wasn't that written in regards to how bad like waterfall a waterfall was? And part of it was like estimating story points. So I feel like I was inspired by that to go, you know, just give them the worst possible estimate, try to copy as much from other projects and see how long they took. And then try and find optimizations so that even if it's 15 percent shorter, it's, you know, we also can use we also use story like poker cards. Those are nice. For one thing, it's very tactile, so you can just go with the cards, but also to you. It's like a way to like, raise disagreements because I think that's that's like the hard part is like sometimes people have information that kind of will help you speed up stuff.

Speaker2 1:04:21

But like you need

Speaker8 1:04:22

To kind of give them the opportunity to go, no, you're wrong because x y z. So story cards like the poker cart, agile poker cards, whatever, those are great full on agile process. Not so much, but agile poker cards. Those are nice so

Harpreet 1:04:40

That there's some aspects of machine learning that maybe you could fit into the agile framework, you know, build the Data pipeline, right? Like, that's pretty tangible.

Speaker2 1:04:51

Like, you know,

Harpreet 1:04:52

Experimentation, probably

Speaker2 1:04:54

Not finding the

Harpreet 1:04:55

Right model. Probably not those things. I guess just time box and move as quickly as possible like I like. Well, say Andrew Ng, that does like those one day sprints just

Speaker2 1:05:09

You can get up and running in

Harpreet 1:05:10

One day, it won't be perfect, but at least we'll give you a skeleton of what to work out, what to elaborate on, I guess. Then what are your thoughts?

Speaker4 1:05:19

Well, a yield most of my time to Greg, because he's the OG here. He'll have possibly the best answer on all of us. But I mean, from my perspective, what I do is break everything up into three different workflows. You've got your Data

Speaker2 1:05:31

Workflow, you've got your research

Speaker4 1:05:33

Workflow and you've got your model development workflow. Your model development workflow is pretty traditional. You know, by that time, you know what you're deploying, you have a pretty good idea of what you're doing. And so that's probably the most predictable side of this. But when you talk about like a data gathering activity, how long is it going to take you to find the data that you need for a particular project? I don't know. I don't even know how to estimate that, you know, it could take, you may

Speaker2 1:06:00

Never find it. You may not

Speaker4 1:06:02

Be able to gather it, you know, so there's there are going to be processes that you can scope with more traditional, more accurate

Speaker2 1:06:11

Than there are. Way, way,

Speaker4 1:06:13

Way uglier ones, which you have to create a different type of framework altogether

Speaker2 1:06:17

Where

Speaker4 1:06:18

You have gated reviews. You know, like you were talking about that one day cycle or one week cycle where you just see how far somebody can get in a week, present it to the rest

Speaker2 1:06:27

Of the team, have something from product

Speaker4 1:06:29

Management

Speaker2 1:06:29

There. Maybe have

Speaker4 1:06:30

Somebody who's a stakeholder, you know, who may be a voice, the user or something like

Speaker2 1:06:34

That and just say,

Speaker4 1:06:36

Hey, this is how far I got. What do you think? Should we do another week on this? And then you can have a conversation and the business can make a decision. We're going to keep going or not kill it next one. And now I yield my time to Greg, who's going to rule this answer

Harpreet 1:06:50

And go for it.

Speaker7 1:06:52

Well, my my focus is going to be about how do you. T-shirt, size pauses, deployment, experimentation. So if you think about a recommendation system. There are some tests that you need to make to make sure that your models are. Behaving or to collect data about whether your models are doing things as expected, and for that, you need to figure out how long it will take you to collect the right data for it. And one good approach to do so is to when you're designing your experiment, you have to split your control in your test at the level of testing. So for example,

Speaker2 1:07:42

You're checking whether your recommendation

Speaker7 1:07:46

System is giving you more sales. And do you want to put that split at the time where the person clicks by? Or do you want to split your population at the time the person enters the website? And for these two stages, you will need different sizes of population, so you have to really understand what is the traffic on your website right now that will give you that population. And obviously, you can have 50000 visits coming in when only 10 percent actually clicks the button to buy. So when you put that split of control and test at the click button, then you don't need as much sample population to perform your experimentation versus putting it at the entrance to at the entrance. You may require you to have 50000 times to one for the control, one for the test. So it will take you more time. And then other considerations to take is the model that I want to test. How much data do you need to ingest in it? What kind of models? So if you're for a recommendation system, you are having this funnel

Speaker2 1:09:02

Approach where your

Speaker7 1:09:04

Filtering from the top down, you may be using a linear model, then you use some excuse and then down there you have a more sophisticated one. But the neural net, these models require a different amount of data and then the time for training and all that stuff. So you have to understand the technical requirements of all of these and understand the size of Data that you need to ingest. So all of these factors and to how do you design a good experiment that will minimize the time it takes to collect the data to confirm whether your newly launched feature is doing the job or not?

Harpreet 1:09:40

Excellent. Thank you so much, Greg Davinia. Any thoughts here?

Speaker3 1:09:48

I was just typing in the chat, but I can go ahead and say it here. Yeah. So I interpreted the question from a methodological perspective, which is how do I communicate to my product owner, my scrum master? Whatever the term du jour is for the person who's running the project, how do I communicate to them the particulars of what the nature of the work is vis a vis the methodology? So for example, if you use Classic Scrum, you need a product owner first and foremost that can define for you up front. Hey, I want you to develop a machine learning model to enhance the product or do something that's cool, and I need that product owner to define what is success, right? And I'm going to use accuracy. Please don't shoot me for I'm just using that as a moniker for success. It could be sensitivity, specificity, whatever, right? F one score or whatever. But I mean, he's accuracy, so I need an accuracy of at least 80 percent. And then what I need as the data scientist, the machine learning developer, whatever the term is from what I do, I then also need the ability within the methodology to

Speaker2 1:10:53

Say, OK, look, that's my

Speaker3 1:10:54

Goal. However, I can't guarantee you that I'm going to be able to deliver that unless you allow me to do what is frequently referred to as a spike, which is a particular type of exploratory work effort within an agile project management framework. And if you're doing classic scrum sprints, we're supposedly 30 days. But if you're XP ish, then you do two weeks, whatever it might be. Hopefully you've got enough time for the duration of your

Speaker2 1:11:19

Spike to do the

Speaker3 1:11:21

Initial perceived initial valuation to say, Look, yep, maybe this is possible product owner that we can do this with additional follow on work

Speaker2 1:11:29

Or nope, it's

Speaker3 1:11:31

Not looking

Speaker2 1:11:32

Too good. So what you need to

Speaker3 1:11:33

Do is you need to be able to incorporate this idea of uncertainty within the methodological framework of how you do project delivery, because that's the reality. Just because you want to build an awesome machine learning model doesn't mean you're going to be able

Speaker2 1:11:44

To do it, but you won't know

Speaker3 1:11:46

Until you try. So you need to have the you need to have the tools in place from your project management perspective to allow you to do that. I've worked on

Speaker2 1:11:54

Teams where we had spikes

Speaker3 1:11:56

That was a particular type of sprint deliverable

Speaker2 1:12:00

For someone who

Speaker3 1:12:02

Did work that wasn't expected necessarily to return back actual working

Speaker2 1:12:07

Code. The idea

Speaker3 1:12:08

Was to gather information, and that's the first thing that you need to start with if you're going to be building a machine learning

Speaker2 1:12:13

Models part of a product development

Speaker3 1:12:15

Methodology.

Harpreet 1:12:18

Awesome. Thanks so much, Dave Coaster. Hopefully that answered your question. Doesn't look like there any other questions coming in on LinkedIn or YouTube or anyone, anywhere else.

Speaker2 1:12:30

So we'll go ahead and wrap up. Guys, thank you so much

Harpreet 1:12:32

For hanging out. Be sure to check

Speaker2 1:12:34

Out the podcast episode that

Harpreet 1:12:36

Was released just today. Dr. Joe Perez Next. We got a big

Speaker2 1:12:41

Week,

Harpreet 1:12:42

A few events. I'll be doing the comet. The office hours on Wednesday, this Wednesday, we've got we're talking all about Data. Understanding Data, validating Data version, Data, Data pipelines, Data governance, all that stuff that a panel discussion with good friend of the show, Matt Belleza will be there. We'll also have Jimmy Whitaker, who's a dev advocate, over at pachyderm. Then we're

Speaker2 1:13:09

Also going to have Dr.

Harpreet 1:13:11

Abe Gong, CEO of Super Conductive. That's a team behind great expectations,

Speaker2 1:13:17

So it'll

Harpreet 1:13:18

Be a great session, so definitely tune in for that. Questions are welcome. Guests are encouraged to come in and join us.

Speaker2 1:13:27

You know, it'll be a lot of fun.

Harpreet 1:13:29

Then the following week, I've got a panel discussion with three other awesome

Speaker2 1:13:33

Individuals people who I

Harpreet 1:13:35

Truly, truly look up to respect and admire. We got

Speaker2 1:13:39

My good

Harpreet 1:13:40

Friend and, you know, I think you all know her.

Speaker2 1:13:42

Jonathan Tully, she'll be

Harpreet 1:13:43

On the on the comet office. We're going to be talking about experiment management, so we got shot independently. Susan Shou Chang Give you guys know Susan. She's epic. She's awesome. Follow her on LinkedIn and then w Ronnie Huang, who used to build laser guns and now is a research scientist at Google. So that's going to be an epic one too. So, yeah, I'm pumped for the remainder of the comet.

Speaker2 1:14:09

Office hours, just

Harpreet 1:14:11

Like for panel discussions lined up over the next few weeks. So it's going to be a lot of fun. So hopefully you guys could tune into that. As usual, a bunch of cool podcast episodes being released in the next few weeks, you're going to have to stay tuned and and see what's up with that. Lot of good stuff happening. And then what else has happened this weekend? But like I said, presentations as well, I'll be

Speaker2 1:14:32

Presenting with

Harpreet 1:14:34

So talked about the panel discussion and then I'm also doing a webinar with Pachyderm on Wednesday, and we're going to be talking about future proofing your ops

Speaker2 1:14:43

Stack.

Harpreet 1:14:44

And then also on Thursday, doing a webinar with cognitive lessons from the field and building your ML apps strategy. So a lot of talk about MLPs, and the more I learn about apps like, the less I know that it's just a massive broad field, so I'd be sure to join in on those if you can interpret.

Speaker7 1:15:10

Yeah. The question is, is this something you won't be doing? Yeah.

Harpreet 1:15:16

I will not be resting. That is for sure that that is for sure, man. Yeah, definitely. Pact for the next few

Speaker2 1:15:26

Weeks

Harpreet 1:15:27

And hopefully making it home to California sometime in February. So I'll be in Sacramento at some point, mid-February. So, Vin, you know, I'd definitely holler at you. Anybody else that is in or around the Northern California zone? Get at me. You know, Sacramento is home, but I'll be spending some time in San Francisco as well. My sister lives out there, so I'll be out there too. So hoping to connect with the, you know, Mickey Cohen and some other friends, maybe even Marc. Guys, take care. Have a good rest of the weekend. Remember you got one life on this planet. Why not try to do some big cheers, everyone?

Download Transcript

Episode Comments

Redirecting to The Employable Data Scientist...