13:42
and confidence and our executive team. I was overseas with my family but returned as quickly as possible. When I learned of the outage and Michael has kept me updated throughout. I have been in contact with Minister Wells throughout the incident and I also spoke with the Prime Minister this morning to provide him with an update directly.
14:17
We have let our customers and Australians down and for that I am deeply sorry. I understand how much Australians rely on us to keep in touch, to work and stay safe. I also understand the broader impact on the community when services go down from things like payments to transport. It's extremely frustrating and disruptive when services aren't available.
14:50
and I am sorry for the impact that this has had on so many people. We take trust in triple zero extremely seriously and it's our responsibility to do everything in our power to make sure calls are answered and transferred immediately. We understand what caused the issue and we will complete our investigation into the actions needed to prevent it from happening again.
15:23
you have my commitment on that. Networks are large and complex and we invest significantly in the resilience of our network. When something goes wrong, we're committed to taking accountability, giving people clear information and fixing issues as quickly as possible. I want to thank our customers and Australians for their patience as we have worked through this.
15:56
I also want to thank everyone across Telstra who has worked around the clock to get to the bottom of the issue and address it and support our customers. I'll now hand over to Michael to update on the current situation. Thank you and as I shared yesterday we implemented a solution that has addressed the impact affecting some calls to triple zero. Our backup systems and welfare checks remain in place however we've not needed to complete
16:28
welfare checks since that solution was implemented. Australians can feel confident in calling triple zero. While the broader outage was resolved yesterday, we've continued to work closely with a small number of our enterprise customers on flow-on impacts to their services, including the Australian Rail Track Corporation. There is a report of a death of a person in South Australia on Wednesday, and our thoughts are with the person's family and loved ones at this incredibly difficult time.
17:02
We've been assisting South Australian authorities in their investigations, including whether there is any connection to Wednesday's outage. We've conducted a detailed review of our network records using the information provided to us, as well as a number of services associated with the same address. To date, we can see no record of calls from those numbers accessing Telstra's mobile network to call 000 and more broadly,
17:31
No record of any calls from those numbers to the 000 platform. We've also confirmed there were no active outages affecting the local area at the time and our records show good mobile signal strength at that location. We can see that a related call was successfully made to 000 from another number which was connected and transferred to the relevant emergency services, emergency authority correctly. So we will continue to work closely with South Australian authorities
18:02
providing information that may assist them and our thoughts remain with the person's family and loved ones. I want to reiterate Vicky's thanks to customers and Australians for their patience in resolving these issues. We know we let customers down and we'll now step back and look at all aspects of this complex issue so we can prevent it from occurring again and make our network stronger. So Vicky and I will now take questions. Many may think you've played down this chaos right from the outset.
18:34
Initially we had 36 people who made triple zero calls, didn't work, and now it's blown out to 600. People want to know, is their services, is everything back to normal? - So yeah, we believe everything is back to normal. Obviously we monitor our network in a lot of detail. We think Australians can be confident in triple zero and can be confident in their services. So we believe we have fixed everything.
19:04
The event was emerging, we shared the information that we knew at the time, there was no attempt to downplay anything and we communicated. So it just gradually got worse? It absolutely gradually got worse. As the time synchronisation propagated out through the network,
19:22
the situation got worse and now we believe we have addressed all of those issues and we're seeing... The South Australian matter, the case there is that the South Australian Police Commissioner has just made an announcement that the husband of this woman made a call to 000. He failed. He then managed to find another way to call 000. The person, his wife, had passed away. So you ring 000, you want action straight away. They didn't get it.
19:52
So what the information that we have at the moment is the information that I just shared. We are continuing our investigations, of course. Is that what I've just said to you? Is that what you're putting? No, what we have is we have no record of that call and I would refer that back to the South Australian Police and we will continue to work with the authorities there. Some really touched the police about the circumstances concerning that call. You just shared some information to that point. Do we know who made the call?
20:24
That is a matter that has been dealt with by South Australian authorities. We're assisting with their investigations with technical data that we can provide them at this stage. So Vicky, can I just ask, how did you first find out about this? And could you kind of run us through, give us a timeline of what you first did and how you managed this remotely over the past two days? Yes, so it was around, it was the early hours of Wednesday morning, Sydney time. What time of day was that? It was around 7am Sydney time. For you?
20:53
For me it was in the evening. I first became aware when I saw a missed message on Teams and a voicemail was left for me by a head of our operations team. I immediately called him back on Teams. So we spoke within minutes of him missing.
21:16
So the outage from the information we have, it was somewhere around 4:20am on that Wednesday morning. Let me speak to that because I think it is a very important point. This issue of time synchronisation in our network
21:33
That's when the first issue occurred. That does take time to propagate across a very large network. Our teams, our technical teams, were working on it almost immediately. So within minutes. Teams operationally were working, were identifying, and it was getting worked. It wasn't that this event had happened and no one was aware of it.
21:57
To the question earlier about do things move around, they do. In this situation, this had to propagate through an entire network with a lot of different nodes, servers, components in that network.
22:11
alongside then more and more customers coming online and doing activity on the network. So our teams absolutely were working on it from the very early hours of Wednesday morning. I was contacted when it hit a certain threshold and it was right around that time that we also then notified key stakeholders like the minister's office and all of those key requirements. So Michael and I became aware
22:40
very close together in time. And then from there in, we have very clear protocols
22:47
If I'm out of the country as CEO, my duties do get delegated very formally and Michael had that delegation. We had very clear crisis management processes and very strong operational controls. All of those processes worked as they should have. So it was around that timeline that I was aware. I then kept in regular contact with Michael and the team
23:14
both while I was on the ground overseas and then through my travel back to Australia. And do you think the processes should be changed so that the CEO of the company can find out about this earlier than almost three hours after the issue occurs?
23:30
So we have really clear processes, as I said, and they have various thresholds in them. Of course, out of any incident, and this incident will be no different, we will conduct very thorough investigations. And one of our focus areas will be, what are the lessons out of this? Are there changes we need to make? But I do want to reiterate, we do have very clear processes around how we manage incidents, and they do escalate through different levels, and this did propagate.
23:58
and once it hit the right thresholds, all of the right parties were notified. Have you ruled out human error yet? Have you ruled out human error regarding this time synchronisation? As you said, it was a chain of events. Where did that stem from? What's the catalyst for that?
24:12
So it started with a software issue. By a human, did they update the software? There was, that's part of the investigation going on. We know that this particular node did restart as work was happening on it and as it restarted this software glitch was triggered. So while a human was working on it, it restarted?
24:34
Yes, which is not unusual in a network of our size. As you're doing work on various components of it, you will have equipment restart. It should be able to restart and go back to the correct time. In this case, it triggered this software issue, which set the clocks back to the incorrect time. What specific immediate measures are you implementing to guarantee that this tight synchronisation software defect can never...
25:00
- So our immediate focus out of this is obviously has been to rectify the immediate issue.
25:10
We are in a position where we are monitoring even more closely all of the components of the network where this issue did occur. We are confident right now the issue has been resolved. We are continuing to work with, as Michael referred to, there are a small number of our enterprise customers that have more sophisticated solutions, things like internet
25:34
internet of things devices where there will need to be field visit to reset devices that's the focus alongside then we will commence our investigation and as Michael has said and as I will reinforce out of that investigation any of those lessons it is absolutely our commitment to make the necessary changes. The ABC reporting has shown that Telstra have been warned about the vulnerabilities of
26:02
So the first thing I'd say is these timing systems are very well known in mobile networks. They are critical. We work with a lot of different universities and professors. We welcome that engagement as part of us making sure we're using best practice. We also work with our peers.
26:26
around the world and with various vendors to make sure we are applying best practice. I have seen some reports this morning. The team tell me we have engaged with that professor that is referenced in that report on a wide range of things and we welcome that engagement. Were you warned? Were you warned? A government-run security centre repeatedly warned you guys about what we were doing.
26:51
- Well, I think we need to get underneath exactly what was part of that conversation. As I understand it right now, it was a broad range of things spoken about, and as I mentioned, these timing synchronisation systems are well known to be critical to a mobile network.
27:09
- I mean, our network is built with a lot of redundancy put in place, and that will be part of this investigation. - Will you and other executives voluntarily offer to surrender your bonuses, given this happened under your law?
27:24
Look, our focus right now has been on getting the issue fixed, making sure all of our customers are back up and running, and then making sure we get the investigation done. We again have very clear processes and governance when it comes to bonuses and remuneration. That process will happen and it will be overseen by our board. Will you accept you oversaw restructuring? There were 100 jobs, just asked a couple of my people.
27:52
With that lack of quality assurance, do you accept that that could have led to what we're seeing now, an outage that was widespread over the nation and potentially is linked to a fatality in South Australia without enough eyes and humans overlooking systems? So my view of what I've seen so far is our people and our processes worked as they should have. As the incident occurred, it was identified.
28:21
and then we've worked to rectify it as quickly as possible. There is no indication that any restructuring of jobs has impacted on this particular issue. As I said, we will complete the investigation and any of those lessons out of that, we will be very transparent and are committed to implementing changes that might be needed. - But that's on the whole council. - How do you expect, Delstra is a premium network, it's marketed itself repeatedly as being the most reliable
28:50
Do you accept that more needed to be done in relation to what was a known risk? Well I think that's a matter for our investigation. So right now we don't have the outcome of that. We invest significant money into resilience of our network and
29:07
but networks are complex and they're not infallible. And so we need to ensure we've got the backup processes and approaches when things do go wrong. In this case, obviously it is unacceptable. And anytime we impact our customers, you know, that's not something we aim to do. And I'm very sorry for the disruption it has caused to so many people.
29:32
We will need to get underneath exactly what caused it and then make sure we take the appropriate steps. Moving forward, how can people trust Telstra in a life or death situation and will people be compensating for this?
29:47
So we understand how much people rely on Telstra services, including the role we play in triple zero. So I would just reinforce our services are working as normal, particularly triple zero. We are seeing it work as normal so people can have confidence in being able to use those services.
30:09
It's, as I said, we will do the investigation. Any changes we deem necessary out of that, we will absolutely implement. Will people be compensated? We talked about confidence 28 hours after this outing. I think Australia will be shocked there's more than 600 missed calls. What does this say about the triple zero custodians, the learnings we were supposed to have after years of this? And as Lynne was saying through...
30:34
Can I just address on the triple zero front, so any issue connecting a triple zero call is a big issue?
30:47
In this case, that is unacceptable, but our backup processes worked. So we identified those calls very quickly. Welfare checks were conducted quickly. So our backup processes did work. Calls did camp on to other mobile networks where they were available. So I agree, triple zero is critical to the country. People need to be able to rely on it.
31:14
Unfortunately, networks are complex and there will be issues. We need to do everything we can to try and minimise those issues, but when they do happen, we must have the processes in place to back that up. And in this case, those processes worked. Calls did camp on where there were other mobile networks available. And we did conduct our welfare checks in a timely manner.
31:41
So this question is about reliability. Telstra has this reputation as a premium, most reliable tariff in the country. Do you think this has harmed that? And if so, what can you tell customers now to make them stay? Will you be considering lowering the price of your plans, which have risen quite considerably under your leadership? What's keeping customers staying with Telstra if something like this could happen?
32:04
So the first thing I'd say is again, apologise to our customers who were impacted through this incident. We know we've got a job to do to make sure we rebuild that trust with our customers and that comes through our actions and through delivering for them and that's absolutely what we will be focused on. When you were first notified of the issue, you said you were communicating via Teams. Is it because you couldn't call?
32:30
It's because the team member who was contacting me was one of the devices that was, he was impacted at that point. - So you're in the middle of an emergency, you're trying to work out what's going on, you can't communicate with each other? - No we can't, because we have again multiple layers of backup in those situations. We have teams.
32:51
You were a victim yourself, basically, of the breakdown. Well, that team member was, but I was able to get back in touch with him via Teams. Did the person identify the systemic timekeeping failure as a hazard in its legally required risk management program? And if so, why was it not able to prevent or mitigate this error?
33:13
That's probably one I don't know if you know the answer to Michael. Yeah, I think that'll be one that we'll come back on in the investigation. Yeah. However, what I would say is that the issue that occurred was due to a software defect that was unexpected. So it wasn't an expected change. Time synchronisation is critical to the way all networks run and yes,
33:38
it clearly has been a vulnerability. But we'll come back on that investigation. But what I would say is what happened was an unexpected change for a system that should not have done that.
33:52
that seven people hadn't gone through to 000 and still required emergency services. Can you please explain for us how long the distance was between that initial call that failed and then them actually being connected with 000 and can you update us on the outcomes for those seven people? - So we, as of the end of yesterday, around 5:30 or a little bit later, we had 604 welfare checks, as we've talked about.
34:19
374 required follow-up contacts, so the rest responded saying they needed no assistance. This resulted in 177 cases being referred to police, including 144 welfare checks and 26 cases where there was a concern a person may have required assistance, so where the police or the emergency services then offered assistance. From those 26 cases, seven were identified as requiring assistance and their details were passed on to the relevant authorities. That was all done
34:49
relatively quickly. I don't have the exact time of all of those and I'm sure we'll hear further updates from that from the relevant authorities. Last question guys, thank you. So how do you then decide how long it takes to send that text message from what meeting of the call and then referring it to police and what's the time frame for that?
35:09
So the way the process works is we'll send a text, we'll wait for a response, we'll send another text, wait for a response, then we'll start doing the calls. We make a number of call attempts relatively quickly and then we refer them. Will your customers be compensated for this? So we will compensate customers under our normal BAU processes and that's underway and customers can reach out through the normal channels to do that. What's BAU? What's BAU?
35:36
Well, you ring up and you make a, you would make a, customers would make a complaint. Our staff are ready to deal with those. They would talk about the impact that it had on them. And then there would be our normal BAU process for compensation. I've spoken with small businesses that couldn't do charges. I've spoken to taxi drivers who couldn't make any money. What would that compensation look like?
36:00
So as part of our BAU process, we would deal with that on a case-by-case basis with those customers. We will make an effort to make it as simple as possible, and yes, those small businesses can call us. - If you actually can see the irony, you're gonna get your bonuses, you and this break, yet a small business owner
36:18
has to go through a VAU complaint which you haven't explained what VAU stands for, let alone the process of being on the phone for an hour to get some monetary form of compensation. So can you see the jarring gap between how you're talking and how Telstra customers on the ground can't even get their transactions completed? As I said, any time we know customers rely on our services,
36:47
We treat all of our relationships with our customers incredibly seriously. We make every effort to make that as simple as possible. And I would encourage customers to reach out and contact their account managers, the people that they work with in Telstra and call directly. We will be doing everything we can to resolve those issues. - Vicky, final question. - Thank you. Final question. - If you have your salary of more than $6 million, do you think the customers are getting value for money with you as SEO?
37:14
Look, as I said, the focus right now is absolutely making sure we have all customers back up and running, we have our network stable. And as I spoke about,
37:25
We have very clear processes from here. When it comes to accountability, I'm the CEO and so it absolutely starts with me and that will be part of what will be considered once we get through and make sure we understand exactly what happened here and what changes might need to be made and our board will oversee that process. - But you took the value for customers.
37:45
So Telstra has a very long history of investing big amounts of capital and operating costs into delivering
38:09
a very high performing network and the largest network mobile network in the country and Customers choose us because of value for money because of the quality of the experience they get when they're not only on our network But they engage with us whether it's in our stores in our contact centers or in our online environment and I accept that
38:31
We have let customers down who were impacted by this incident and it is on us to rebuild that trust and demonstrate through our actions that we will continue to deliver highly reliable services to them. Thank you guys. We're going to have to wrap it up. Thank you very much for your time. Appreciate it.
