Saturday, March 19, 2016

Week 96 Part 2: Safari Takes Refuge

          Ninety-six safaris is a lot. As we approach our 100th consecutive week of group hypergridding, seems like we're all aware of how far hg has come over the past two years.
DJ Pooh's splendid Honeypot region
          The safari is like a bumblebee. The laws of physics say it shouldn't fly, yet it does. When we began, no grids were set up with mass hypergrid tp arrivals in mind. Even today, many grids  - even big name places, with a good reputation for stability - tremble and even crash when we arrive. You may say, stop doing it. Yet, that is really the whole point of the exercise. Gridhopping alone or with a couple of friends is great too, but what we are here to do is to push the envelope, and raise awareness that splendid isolation, while it can be restful at times, is no way to run a grid on the HG. And we like to think that our relentless insistence on mass arrivals have made things smoother for the lone traveller, and made grid owners more conscientious and given them a greater appreciation for visitors of all dimensions.
          Because life is movement.
          This is part 2 of our 96th Safari. The final hour at DJ Pooh's beach-heavenly dance venue on sim Honeypot, refugegrid was one of music, courtesy of Truelie Telling, dressed to express the festive spirit of St Patrick...  the following evening she headlined at the Maritime Club pass-the-mic event, along with Casias Falta, Greybeard Thinker, Joao Frazao and Nazirah Avro. It was an unprecedented evening at the Maritime. Club owner Stiofain Mactomais had each one perform numbers inspired to a large extent by celtic music, and after two songs, he would switch stream. Each performer was on mic twice, creating a great energy in the room, and pulling together our global community, from Australia to the US, from Venezuela to England, in a unique and wonderful way.   
Truelie Telling greening it up for saint paddy

          Truelie sings for the Safari every month, sharing original numbers about life love and mishaps in the virtual and real worlds, Many times it's the first time the region owner has ever had a live performer on their land, which if you think about it, makes Truelie an extraordinarily flexible and brave performer, as well as a talented one.

          DJ Pooh made us very welcome on her sim, and there's a welcome for all music lovers over there, should you feel like dropping by. You will find the HG Address at the end of this post.
          We were on Refuge Grid to meet grid co-founder Seth Nygard, an IT guy with decades of experience who promised to share some thoughts about the care and feeding of OpenSim. Thank you Lucy Afarensis for suggesting this Safari appointment.
     This is  a really long post, despite editing. Not all the questions that got asked were answered, as you can imagine in this format, it's hard to read all the chat. However, I've tried to include even unanswered questions, because you may have a similar problem, or know the answer. Here goes....
Thirza.Ember: yay here we are! or some of us
Seth Nygard: : Hello! I see the sim busily trying to traverse everyone's inventory so lots of messages
James Atlloud hmm, interesting
Pathfinder Lester: oh boy, i need to clean up my suitcase. it's full of stuff :P
James Atlloud: Mine too
Pathfinder Lester: I pick up so much stuff as I explore the hypergrid
Taarna Welles: Tadaaa. I know most of us knows what it means seeing all the text scrolling in your console :)
James Atlloud: A good inventory filter would be a 'infrequently used' option
Seth Nygard: It is too bad the sim has to work through the entire root for everyone, that wastes a lot of time since we know things outside the suitcase can't be read unless you are on OSGrid or one of the grids running version 1.0 of the HG inventory
Pathfinder Lester: James, good idea
snowbody Cortes: uhm ... difficult...
James.Atlloud: @snow - about inventory filter? probably no datapoint of usage date
Lucy.Afarensis: good thing I did not come as Osgrid Lucy, more than 15000 items in hers
Seth Nygard: There really isn't, but limiting to the suitcase should be doable without much effort That would be a viewer change however most likely
Pathfinder Lester: Seth, are you saying that when you are a hypergrid visitor, the sim you're on has to work through your entire inventory? So if you have a huge inventory and a small amount in the suitcase, you'll still be a huge load on the sim?
James Atlloud: Ooo I haz 3,652
Aime.Socrates: ops
Wizardoz.Chrome: umm haaaaaaaa !
Arriving on Pericynthion, Seth's home sim
Taarna.Welles: Then it would be nice to have 2 Inventories, so you can choose which one you want to use.
Seth Nygard: yes, for visitors, every inventory folder is scanned. Most just get skipped with an "Unable to fetch..." message but the attempt is made for them all. That was supposed to be the purpose of the Suitcase, but the viewer really doesn't know that the other folders are not available until if checks them all. LJ has over 75K in hers so it takes a long time.
James Atlloud: jeepers!
Pathfinder Lester: Seth, i didn't know that. i learn something new every day. :) ty
Taarna Welles: I stopped counting :-)
George Equus: OSG don't use the Suitecase
Seth Nygard: : OSGrid does not use HG Inventory 1.5, so there is no suitcase. That however is also a bit for a security risk but that is their choice in setup
Truelie Telling: well darn, time for an inventory diet 8-<
Thirza.Ember: almost all my francogrid inv is in My Suitcase, probably. i feel bad.. And yet,I do nothing. Now I feel less bad.
Seth Nygard : The server is currently using 25MB/s of upstream data for the ones that are here
Lucy.Afarensis: So would it help to have a safari avie with a small inventory
Seth Nygard : yes, that would help. The more you can reduce what the sim has to do the better it can handle things. Thread handling in general is very poor.
snowbody Cortes: Seth , what's that Unknown User bug that is still there ?
Pathfinder Lester: oh i hate that Unknown User bug
snowbody Cortes: it is weird because not all the people see the same user as Unknown User
Seth Nygard : There seems to be a few causes for that and I am not so sure the devs even understand that is going on. The viewer is also partly to blame since if you look at someone's name tag their name will show, but everywhere else it is unknown. It happens especially for HG visitors their first visit to a sim. Once you have visited and left data is persisted for your account, making lookup much easier subsequent visits.
Thirza.Ember: hmmm... I don't know about that. I'm sometimes unknown on my own sim
Seth Nygard : Yes, I've seen that as well but usually in isolated cases
George Equus: All but Max has names but he is "737ef974-8bb7-4a39-83e3-d4a2f7786b8b" only
Thirza.Ember: George, that's Max's RL name.
George Equus: Ahhh ! :) Would love to hear his mother call for him...
Truelie Telling: she whistles
Lucy Afarensis: I have a sim with 11000 prims and a lot of large textures the safari wants to visit and hold a concert there, will this work ?
snowbody Cortes: lol - prepare your sim for the crash Lucy !
Truelie Telling: it's worked on my sim, Lucy, with 15000 prims
Thirza Ember: yes Truelie but we could only see about 10 of those prims
Truelie Telling: hehe
Seth Nygard : That is a hard one to answer, Usually I wouldn't recommend a large group to a prim heavy region, but with some care it can work. Textures account for a lot of data, I am sure on your many safaris you see all sorts of various issues
Lucy Afarensis: What kind of care Seth ?
Seth Nygard: : try to pace the visitors, so they TP in one at a time and allow things to rezz. There is a lot of data exchanged when anyone first TPs into a region.
Lucy.Afarensis: I can get cloud serve to boost the power of the server, would that help ?
snowbody Cortes: Seth, when a new avatar arrives in a sim, does it change the download policy for the other avatars ? I mean there's some rules inside opensim there ? I was wandering some nights ago, if you arrive last you need a very long time to see someone else so it seems there is no policy.
Seth Nygard : There is some degree of throttling that can be configured but that also appears to be less than ideal. There are cases where I have seen that cause more problems
Truelie Telling: snow, that's what we've seen at the Maritime, the more people, the slower the rezz
snowbody Cortes: yes Truelie !
Seth Nygard... Not just a pretty face
          We also got Seth to speak to the vexed question of arriving on a new grid via the Welcome area. Human nature being what it is, most of us prefer to TP directly to the sim we wantt o visit, rather than going through a clunky stepping-stone process, unless we already know a direct jump is unlikely to work, as in, for example, a lot of Var regions. When it comes to safarying, where each tourist has a package of LMs, I try to avoid making any more jumps than necessary, because it can be time consuimg looking for people who are lost. But Seth is a fan of the 'First jump to Welcome' school of thought. He explains why.
Seth Nygard : What OpenSim has the most trouble with is packet errors or re-sent packets. Too many and that will cause people to crash out of a sim. I have an open Mantis on that one and believe it is a much bigger problem than most think it is. It affects everyone. The first HG TP into a grid has the greatest amount of data, subsequent TPs in the same grid are then handled the same as any other local TP. I also share data for the local flotsam cache between all the high traffic regions. That limits the data that has to be pulled from the gird asset server
Thirza Ember: when we has the safari on Hedonism, almost everyone could only see bits of the exotic dancers... some just torsos, some only half a body... that was an interesting event... is that related to our too hasty or direct tp'ing? Was also the most gruesome safari ever I think
Pathfinder Lester: lol, nightmare fodder?
James.Atlloud: any video?
Thirza Ember: no video, but was kinda weird to see a pair of legs pole dancing by themselves
Welcome sim, RefugeGrid
Seth Nygard : That can also be related to the poor thread handling where data was not sent in the time frame it should have been due to one or more connections holding things up. There is also issues where some packets are sent to a viewer only once. If they are missed then things won't rezz, or animations won't play.
snowbody Cortes: yes Seth, for example dancing balls also stop to work and it seems avatars are frozen for a while, then they start again
Seth Nygard : just an FYI, the sim is still working its way through the inventories for everyone. There is a constant stream of messages like this;
16:31:45 - [WEB FETCH INV DESC HANDLER]: Unable to fetch folder f18aa038-7d9b-9d7c-00c5-e165d3f15323
16:31:45 - [WEB FETCH INV DESC HANDLER]: Unable to fetch folder f18aa038-7d9b-9d7c-00c5-e165d3f15323
… approx 10 every second
Thirza Ember: so we can stop that how - by pruning inventory?
Seth Nygard : Pruning inventory can help for sure. Every folder gets scanned for every user
Thirza Ember: so there is an old wives' tale that if we sit down or dance, it's less likely that a sim will crash... is this true do you think? or just voodoo
Truelie Telling: dancing too, I have been told
Seth Nygard : There is some truth to that. When you are sitting you are no longer part of the physics calculations so have less impact on the server
We all sat on the ground. A few people showed up, said hi all round the group, and slipped away. If you've never been on Safari, it's a constant coming and going of new people and old friends. Ome just drop in to say hi, and then go off to build or whatever. There's never a week goes by that there aren't new people to Friend. You don't have to know people in the group to participate, you don't have to show up for the whole thing. So don't be shy, come and join us one of these weeks.
Taarna Welles: Sorry, my legs became a bit tired. It's heavy if you stand all the time on high heels
Seth Nygard : You need to wear flats on days like this
Taarna Welles: You are right Seth!  
snowbody Cortes: another question: does voice lag the sim ? I suppose not, but I'm not sure
Seth Nygard : Voice should not normally lag the sim, it is not really handled by the sim much. Looking through your inventory can certainly lag a busy sim since all that data is marshalled through the simulator. Group messages also seem to have a large impact on a busy sim and can contribute to lag. The higher your draw distance is set, the more data is needed and that contributes to lad
Thirza Ember : crazy that the suitcase doesn't really do anything to stop lag
Taarna Welles: I thought too, the default inventory would be disconnected. Apparently not.
Seth Nygard: No, the suitcase really only helps protect the rest of the content but is not leveraged to reduce the data demands unfortunately
James Atlloud: Seth, could you imagine assets being saved at 3rd party location such as Google Drive, Amazon S3?
Seth Nygard: Assets could certainly be handled differently, but the simulator would need changes to allow that
Taarna Welles: So for HG the best is to use a traveler avatar, like Thirza often use.
Seth Nygard: An avi with a minimal inventory certainly has much lower impact on things, so yes that can help
Taarna Welles: Even then, HG is so much better if you compare it with two years ago. For me it's always smooth... most of the time.
snowbody Cortes: so true Taarna
James Atlloud: I have to agree with Taarna - I have more good experiences than bad. <knock wood>
Seth Nygard: HG has really not seen much development and what we do have IMHO is more a proof of concept than a total solution. There have been some fixes that help, but a lot more could be done
Taarna Welles: But we all can try to improve the experience though?
Thirza.Ember: i think even if the code has not changed, grid owners have made strides to improve our experience
Seth Nygard: Yes, I think so and it what I want to try to help with where I can. User experience is perhaps the most important. Without it people don't stay.
James Atlloud: Hey Seth, could opensim core use some donations?
Seth Nygard: I think so but no one really knows how to best manage that whole process so it works for everyone. I really want to see more community<>dev communication but that must work openly and in both directions for it to be of any use to anyone
snowbody Cortes: yes Seth that sharing could help and motivate more the devs too
James Atlloud: I really have great hopes for opensimulator and I would be highly motivated to support the devs in some way.
          Ni Lemon arrived just about at this moment.
Thirza.Ember: hey Ni we are talking about how to make regions work better for hg, in chat. You have to get rid of all your inventory, that is OK, right?
Taarna.Welles: We are willingly to share a script which will do it for you
George Equus: There are barrels by the gazebo to dump the inventory into
Seth Nygard: You will find that generally regions running on home servers have the most problems. That is mostly due to the data demands that really do not fit the residential internet model
Thirza.Ember: our rule has always been to just go places. If they crash, they crash. Many surprisingly have not, many that should have been rock solid have dumped us
George Equus: admission fee = Inv contents
Ni Lemon: leave where? do you have a bin?
Sunbeam Magic: true Thirza one never knows
Seth Nygard: If everyone travelled as Ruth and with an empty inventory then things would work best. That however does not give much a user experience when we all like to customize our avis
Thirza Ember: if somebody says 'this region is rock solid = will probably crash when 16 or 17 of us arrive
Lucy Afarensis: rule 1 never brag about your server
Seth Nygard: Unless there is someone with a poor network connection the sim should be able to handle as many avis as we can throw at it
Taarna.Welles: The question is: are you traveling to show your avatar or to see what others made?
George Equus: I found last six months or so my outfit usually works 100 per cent all over the various grids
James Atlloud: yes @taarna
Truelie Telling: so some else's bad connection can hurt the sim?
Seth Nygard: We've had 38 at one time so far with no problems, but there are times were a single user can grind any sim to a halt
Total Sorbet: yes I've noticed this
Seth Nygard: Yes, if there are too many packet retries OpenSim spends all its time servicing that thread and others timeout. That is what I believe is 95% of the crashes that many of you experience in various sims. One user can and does have an impact on every other user and the sim in those cases. This is one of my own open Mantis tickets on that very issue;
Lucy Afarensis: Is it possible to kick theM off before the sim crashes ?
snowbody Cortes: and the sim has any policy to handle that?
Seth Nygard: Throttling was put in place to try to help with that but it can contribute to the problem causing a perfect storm of data. I can usually spot the sings just before it happens but not quck enough to kick out offending users to keep the cascade from happening. I have been collecting a lot of data to try to determine the actual causes
James Atlloud whispers: Need a network speed indicator in the nearby avatar panel
Pericynthion panorama
Taarna Welles: Other question: a few days ago I was HG to another grid, and when I came back I couldn't see a lot of meshes I made, even not my own clothing. Only with the singularity viewer I was able to see the meshes. After I restarted the regions I was able to see them again with my default FS viewer. I have know idea why this happened. Is there someone who tell me why this happened?
Unadecal Masala: that's really interesting. I am experiencing that at the moment in a sim I'm hosting for someone. I get continuous packet retries and they complain about massive lag
Seth Nygard: Without being able to review the logs it is hard to say what may have caused that. If you are seeing a lot of packet resends then you will most definitely see lag and other issues
Unadecal Masala: ok
James Atlloud : This is so good to know, I'm really appreciating this informaton.
Seth Nygard: I have fine tuned the linux kernel that runs on my servers to help optimize things but the problem is really how OpenSim handles data and its threading mechanism
Sunbeam Magic: I keep coming across this issue where a new outfit keeps turning in to 'new shirt' and makes duplicates over and over and I can't delete them ... have you heard of that before ? Is it me or the designer ?
James Atlloud whispers: I have that shirt issue @sunbeam
Thirza Ember: so really the sim should kick a 'bad' avie? for the good of the group?
Lucy Afarensis: Does it help to restart a sim before a crowd is expected ?
Seth Nygard: Yes it can help. I do tend to do a hard restart before a party or large group. There are many things that do not get cleaned up in memory otherwise.
Thirza Ember: restart = first law of safari
Seth Nygard: I have however also been able to run the same region for 4-5 weeks with no restarts and not had issues.
Taarna Welles: Oh dear, I have to go offline. Thank you Seth for more understanding about some issues. Sleep well and until the next Safari!
Sunbeam Magic: should one restart weekly ?
Seth Nygard: There is a reason even SL has restart Tuesdays. I know some people that do daily restarts
Sunbeam Magic: wow daily doesn't seem right lol
Seth Nygard: If possible watch the memory and thread use. You will see things start to increase and that is a sign that a restart could be in order.
snowbody Cortes: probably it should depends on the amount of textures and other stuff you upload too
Sunbeam Magic: we get a whole lot of traffic. Is it true to try and keep scripts total to 18.0 or less per region
Seth Nygard: Yes, so many things are dependent on what you do and how much traffic is seen . Scripts are the biggest culprit of those issues. Scripts on visitors, even when not enabled also contribute
snowbody Cortes: also if they are not enabled ?
Sunbeam Magic: I didn't know that Seth
Seth Nygard: Many scripts are very poorly behaved. You will see sometimes the script count go negative. The count is not very accurate once you have visitors come and go - scripts on each avi get loaded even if they are not allowed to run.
          And just like that, our time was up. No doubt about it, we will be coming back to talk to Seth again. There is so much to learn from each other. That's Safari!
HG Address
Visit refugegrid Welcome sim to explore the regions open to the public
Welcome sim, Refuge Grid

No comments:

Post a Comment