AfterChange Collection Hook shows Validation Error that prevents updating another collection with created doc.id

Link to reproduction
No response
Describe the Bug
New bug introduced in Payload 2.0 where the afterChange collection hook does not have knowledge about the created doc. I see a Validation Error when I try to update another collection with the recently created document id. Error states it is an invalid selection.
To Reproduce
This flow worked in my previous version before upgrading from 1.13.2 to latest payload 2.3.1.
Background:
I have a stores collection with a field name: "foods" with type: "relationship" and relationTo: "foods".
I have a foods collection with a field name: "storeId" with a type: relationship" and relationTo: "stores"
After I create a new food, I have an afterChange hook "addFoodToStore". I get the doc.id (since the food was created and I have the id) and push it to the newFoods array where I will payload.update collection "stores" with the newFoods.
However, I keep getting the validation error:
[00:08:22] ERROR (payload): The following field is invalid: foods
    err: {
      "type": "ValidationError",
      "message": "The following field is invalid: foods",
      "stack":
          ValidationError: The following field is invalid: foods
              at beforeChange (.../node_modules/payload/src/fields/hooks/beforeChange/index.ts:60:11)
              at processTicksAndRejections (node:internal/process/task_queues:95:5)
              at updateByID (.../node_modules/payload/src/collections/operations/updateByID.ts:230:18)
      "data": [
        {
          "field": "foods",
          "message": "This field has the following invalid selections: \"656e69f69002dcffc8e7847e\", "
        }
      ],
      "isOperational": true,
      "isPublic": false,
      "status": 400,
      "name": "ValidationError"
    }
[00:08:23] ERROR (payload): ValidationError: The following field is invalid: foods
    at beforeChange (.../node_modules/payload/src/fields/hooks/beforeChange/index.ts:60:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at updateByID (.../node_modules/payload/src/collections/operations/updateByID.ts:230:18)
       "field": "foods",
          "message": "This field has the following invalid selections: \"656e69f69002dcffc8e7847e\", "
        }
      ],
      "isOperational": true,
      "isPublic": false,
      "status": 400,
      "name": "ValidationError"
    }

If I move back to Payload version 1.13.2, the issue is no longer there. I have moved to Payload 2.0 but this is a major change that prevents me from adopting.
My analysis is that when req.payload.update is called to update the stores collection with newFoods, the food actually has not been created. I even tried running req.payload.update food collection with the doc.id and it said id did not exist.
Therefore, afterChange is not functioning after the document is created as it states in the docs even though there is an id for the doc object. After the error appears, I do not see a new food record so I suspect must be something wrong w/ the afterChange hook.
Payload Version
2.3.1
Adapters and Plugins
No response

Hey @miiketran this appears to be working for me, so I wonder if there's something else going on with your config. I wasn't able to reproduce this, but I did create a branch with a working proof of concept. Here's what my hook looks like:
{
  // Foods.ts
  // ...
  hooks: {
    afterChange: [
      async ({ doc, req }) => {
        if (doc.store) {
          try {
            const storeID = doc.store && typeof doc.store === 'object' ? doc.store.id : doc.store

            // Add food to store
            const store: Store = await req.payload.findByID({
              collection: 'stores',
              id: storeID,
            })

            const newFoods = [...(store?.foods || []), doc.id]

            await req.payload.update({
              collection: 'stores',
              id: storeID,
              data: {
                foods: newFoods,
              },
            })
          } catch (error) {
            req.payload.logger.error(error)
          }
        }
      },
    ],
  },
  // ...
}

Would you be able to provide a code snippet from your project, or more context? Or better yet, pull down this branch and try to create a reproduction in the _community directory.

I too am experiencing this on 2.4.0 with postgres. You can see this in action at my test repo.
I have an employees collection and a sorting global. I've created a hook that SHOULD populate the sorting collection with a new employee relationship when a new employee is created. Unfortunately, when executing the hook, the following error is thrown:
error: insert or update on table "sorting_rels" violates foreign key constraint "sorting_rels_employees_id_employees_id_fk"
    at ... {
  length: 312,
  severity: 'ERROR',
  code: '23503',
  detail: 'Key (employees_id)=(19) is not present in table "employees".',
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: 'public',
  table: 'sorting_rels',
  column: undefined,
  dataType: undefined,
  constraint: 'sorting_rels_employees_id_employees_id_fk',
  file: 'ri_triggers.c',
  line: '2608',
  routine: 'ri_ReportViolation'
}

From what I can tell, it seems like the hook is trying to update the global before the new employee record is saved to the database.
For some additional context, my hook is defined like so:
export const addToSortedHook: CollectionAfterChangeHook = async ({
  doc,
  operation,
  req,
}) => {
  // Only run this hook when creating a new employee
  if (operation === "create" && doc) {
    try {
      const { id } = doc;

      const employeeId = parseInt(id);

      const { employeeOrder } = await req.payload.findGlobal({
        slug: "sorting",
        depth: 0,
      });

      // BUG: It currently appears that the employees table has not been updated with
      // the new employee record at the time this hooks is run.

      await req.payload.updateGlobal({
        slug: "sorting",
        data: {
          employeeOrder: [...employeeOrder, employeeId],
        },
        overrideAccess: true,
      });
    } catch (error) {
      console.log(error);
    }
  }
};

My employee collection can be seen here:
const Employees: CollectionConfig = {
  slug: "employees",
  fields: [
    slug("name"),
    {
      name: "name",
      type: "text",
      required: true,
    },
    {
      name: "createdBy",
      type: "relationship",
      relationTo: "users",
      admin: {
        position: "sidebar",
        readOnly: true,
        condition: (data) => data.createdBy,
      },
    },
    {
      name: "title",
      type: "text",
      required: true,
    },
    {
      name: "bio",
      type: "richText",
    },
    {
      name: "image",
      type: "upload",
      relationTo: "media",
    },
  ],
  hooks: {
    beforeChange: [setCreatedByUserIdHook],
    afterChange: [addToSortedHook],
  },
  access: {
    read: () => true,
    create: isAdmin,
    update: isAdmin,
    delete: isAdmin,
  },
  admin: {
    useAsTitle: "name",
    defaultColumns: ["name", "title"],
  },
  versions: {
    drafts: true,
  },
};

And my global Sorting config is defined like so:

const Sorting: GlobalConfig = {
  slug: "sorting",
  fields: [
    {
      type: "tabs",
      tabs: [
        {
          label: "Employees",
          fields: [
            {
              type: "relationship",
              relationTo: "employees",
              name: "employeeOrder",
              hasMany: true,
            },
          ],
        },
      ],
    },
  ],
  access: {
    read: () => true,
    update: isAdmin,
  },
  admin: {
    group: "Site Settings",
  },
};

Hi @jacobsfletch , thanks for looking into it. I'm having trouble running the _community directory locally so I'm not able to test.
To confirm, in your working proof of concept, you say you're successfully able to update the stores collection with the newFoods?  I don't know if there is more insight I can provide besides me receiving a validation error when trying to update stores with newFoods.
I believe @creekdrops is illustrating the same issue as mine. We can switch to looking into their issue and I think it may also resolve mine.

I too am experiencing this on 2.4.0 with postgres. You can see this in action at my test repo.
I have an employees collection and a sorting global. I've created a hook that SHOULD populate the sorting collection with a new employee relationship when a new employee is created. Unfortunately, when executing the hook, the following error is thrown:
error: insert or update on table "sorting_rels" violates foreign key constraint "sorting_rels_employees_id_employees_id_fk"
    at ... {
  length: 312,
  severity: 'ERROR',
  code: '23503',
  detail: 'Key (employees_id)=(19) is not present in table "employees".',
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: 'public',
  table: 'sorting_rels',
  column: undefined,
  dataType: undefined,
  constraint: 'sorting_rels_employees_id_employees_id_fk',
  file: 'ri_triggers.c',
  line: '2608',
  routine: 'ri_ReportViolation'
}

From what I can tell, it seems like the hook is trying to update the global before the new employee record is saved to the database.
For some additional context, my hook is defined like so:
export const addToSortedHook: CollectionAfterChangeHook = async ({
  doc,
  operation,
  req,
}) => {
  // Only run this hook when creating a new employee
  if (operation === "create" && doc) {
    try {
      const { id } = doc;

      const employeeId = parseInt(id);

      const { employeeOrder } = await req.payload.findGlobal({
        slug: "sorting",
        depth: 0,
      });

      // BUG: It currently appears that the employees table has not been updated with
      // the new employee record at the time this hooks is run.

      await req.payload.updateGlobal({
        slug: "sorting",
        data: {
          employeeOrder: [...employeeOrder, employeeId],
        },
        overrideAccess: true,
      });
    } catch (error) {
      console.log(error);
    }
  }
};

My employee collection can be seen here:
const Employees: CollectionConfig = {
  slug: "employees",
  fields: [
    slug("name"),
    {
      name: "name",
      type: "text",
      required: true,
    },
    {
      name: "createdBy",
      type: "relationship",
      relationTo: "users",
      admin: {
        position: "sidebar",
        readOnly: true,
        condition: (data) => data.createdBy,
      },
    },
    {
      name: "title",
      type: "text",
      required: true,
    },
    {
      name: "bio",
      type: "richText",
    },
    {
      name: "image",
      type: "upload",
      relationTo: "media",
    },
  ],
  hooks: {
    beforeChange: [setCreatedByUserIdHook],
    afterChange: [addToSortedHook],
  },
  access: {
    read: () => true,
    create: isAdmin,
    update: isAdmin,
    delete: isAdmin,
  },
  admin: {
    useAsTitle: "name",
    defaultColumns: ["name", "title"],
  },
  versions: {
    drafts: true,
  },
};

And my global Sorting config is defined like so:
const Sorting: GlobalConfig = {
  slug: "sorting",
  fields: [
    {
      type: "tabs",
      tabs: [
        {
          label: "Employees",
          fields: [
            {
              type: "relationship",
              relationTo: "employees",
              name: "employeeOrder",
              hasMany: true,
            },
          ],
        },
      ],
    },
  ],
  access: {
    read: () => true,
    update: isAdmin,
  },
  admin: {
    group: "Site Settings",
  },
};

same

@jacobsfletch I was able to reproduce the error using your example
I created a repo using your config and the create payload cli. I also updated some of the package versions
The repo: https://github.com/timoconnellaus/payload-reproduce-error
[05:31:10] ERROR (payload): The following field is invalid: foods     err: {       "type": "ValidationError",       "message": "The following field is invalid: foods",       "stack":           ValidationError: The following field is invalid: foods               at beforeChange (/Users/tim/repos/payload-reproduce-error/node_modules/.pnpm/payload@2.4.0_typescript@4.8.4_webpack@5.89.0/node_modules/payload/src/fields/hooks/beforeChange/index.ts:60:11)               at processTicksAndRejections (node:internal/process/task_queues:95:5)               at updateByID (/Users/tim/repos/payload-reproduce-error/node_modules/.pnpm/payload@2.4.0_typescript@4.8.4_webpack@5.89.0/node_modules/payload/src/collections/operations/updateByID.ts:230:18)       "data": [         {           "field": "foods",           "message": "This relationship field has the following invalid relationships: [object Object] 0"         }       ],       "isOperational": true,       "isPublic": false,       "status": 400,       "name": "ValidationError"     }

@jacobsfletch I was able to reproduce the error using your example
I created a repo using your config and the create payload cli. I also updated some of the package versions
The repo: https://github.com/timoconnellaus/payload-reproduce-error
[05:31:10] ERROR (payload): The following field is invalid: foods err: { "type": "ValidationError", "message": "The following field is invalid: foods", "stack": ValidationError: The following field is invalid: foods at beforeChange (/Users/tim/repos/payload-reproduce-error/node_modules/.pnpm/payload@2.4.0_typescript@4.8.4_webpack@5.89.0/node_modules/payload/src/fields/hooks/beforeChange/index.ts:60:11) at processTicksAndRejections (node:internal/process/task_queues:95:5) at updateByID (/Users/tim/repos/payload-reproduce-error/node_modules/.pnpm/payload@2.4.0_typescript@4.8.4_webpack@5.89.0/node_modules/payload/src/collections/operations/updateByID.ts:230:18) "data": [ { "field": "foods", "message": "This relationship field has the following invalid relationships: [object Object] 0" } ], "isOperational": true, "isPublic": false, "status": 400, "name": "ValidationError" }

I was able to recreate your issue. To solve it, you should pass the ID's of the foods, you can by fetching stores like so:
const store: Store = await req.payload.findByID({
  collection: "stores",
  id: storeID,
  depth: 0
});

or you could filter the foods down to just their ID's and then spreading the ID's when merging. You would also likely want to ensure the ID of the store you are editing does not already exist in the stores list.

Still reproducible when running "@payloadcms/db-postgres": "^0.7.0" & "payload": "^2.11.1".
Trying to run an afterChange hook to create an object from another (non-global) collection which has a reference to the first collection and get the following error:
error: insert or update on table "test_rels" violates foreign key constraint "tests_rels_users_id_users_id_fk"

Had the nearly same problem as described. But for me the request to update another collection never finished - no error, not response...nothing.
Switched from postgres to mongodb and the error was gone.

(Using postgres as well) We've found that if you don't await the update/create call in the afterChange hook, it seems to work as intended. However, this could still indicate a timing issue and might still result in an error if the timing is slightly off between the two operations. This is also not a full fix if you need the result for further operations of course

I've run into the same problems intermittently. The behavior seems to alternate between the ValidationError, or the pg constraint error. I was able to fix it with two updates:

Pass the req as an option to payload operations within the hook
Cast doc.id to a number before passing it as value back to payload (if you're using postgres)

const hook: CollectionAfterChangeHook  = async ({ doc, collection, req }) => {
  const data = await req.payload.create({
    collection: 'pages',
    data: {
      post: {
+       value: Number(doc.id),
-       value: doc.id,
        relationTo: collection.slug
      }
    }
+   req,
  })

  return doc
}

Passing req threads the request into the same transaction as the initial change operation, fixing the race condition. The option was introduced in #5068 and I already found a related regression in the search plugin -- I bet this subtly broke a number of hooks.
As for casting the id to a number... there must be some edge case in core where it's sometimes returned as a string. I gave up trying to find the source of that one. Might be related to #5794.
"@payloadcms/db-postgres": "^0.7.1",
"payload": "^2.12.1"

(Using postgres as well) We've found that if you don't await the update/create call in the afterChange hook, it seems to work as intended. However, this could indicate a timing issue and might still result in an error if the timing is slightly off between the two operations. This is also not a full fix if you need the result for further operations of course

This seems to work. Using postgres as well

(Using postgres as well) We've found that if you don't await the update/create call in the afterChange hook, it seems to work as intended. However, this could indicate a timing issue and might still result in an error if the timing is slightly off between the two operations. This is also not a full fix if you need the result for further operations of course

Thanks for this! was having the same issue as everyone else here.  got it working by removing await, but yeah this does pose an issue if we need to do something more complicated..
To share details on my side:
Im using Postgres, payload 2.30.1 with db-postgres 0.8.9.
In my debugger, I noticed the expected new record does not show at all in the database when I put a breakpoint in both the afterChange and afterOperation hooks, and get the following error (which makes sense given that the record does not yet exist at this point):
Error setting up initial community organizer: error: insert or update on table "community_organizers_rels" violates foreign key constraint "community_organizers_rels_communities_fk"


[api_error] Log logged in, reconnecting in 900

i get this infor from linux terminal -->2016-08-09 13:03:02,949 [       cli] [INFO] PokemonGO Bot v1.0
2016-08-09 13:03:02,953 [       cli] [INFO] Configuration initialized
2016-08-09 13:03:02,953 [pokemongo_bot.health_record.bot_event] [INFO] Health check is enabled. For more information:
2016-08-09 13:03:02,953 [pokemongo_bot.health_record.bot_event] [INFO] https://github.com/PokemonGoF/PokemonGo-Bot/tree/dev#analytics
2016-08-09 13:03:02,960 [requests.packages.urllib3.connectionpool] [INFO] Starting new HTTP connection (1): www.google-analytics.com
2016-08-09 13:03:03,303 [PokemonGoBot] [INFO] [set_start_location] Setting start location.
2016-08-09 13:03:03,303 [PokemonGoBot] [INFO] [x] Coordinates found in passed in location, not geocoding.
2016-08-09 13:03:03,304 [PokemonGoBot] [INFO] [location_found] Location found: 59.333409,18.045008 (59.333409, 18.045008, 0.0)
2016-08-09 13:03:03,304 [PokemonGoBot] [INFO] [position_update] Now at (59.333409, 18.045008, 0)
2016-08-09 13:03:03,304 [PokemonGoBot] [INFO] [login_started] Login procedure started.
2016-08-09 13:03:04,307 [pgoapi.auth] [ERROR] Could not retrieve token: Your username or password is incorrect. You have 4 attempts left before you will be locked out of your account for 15 minutes.
2016-08-09 13:03:04,308 [PokemonGoBot] [INFO] [login_successful] Login successful.
2016-08-09 13:03:04,309 [PokemonGoBot] [INFO] [api_error] Log logged in, reconnecting in 900
but it never connects
this is the basic config    ###file...
{
"auth_service": "ptc",
"username": "<<<<<<<<",
"password": "<<<<<<<<<<<<<<<",
"location": "59.333409,18.045008",
"gmapkey": "XXXXXXXXXXXXxx",
"libencrypt_location": "/XXX/PokemonGo-Bot",
"tasks": [
{
"type": "HandleSoftBan"
},
{
"type": "CollectLevelUpReward"
},
{
"type": "IncubateEggs",
"config": {
"longer_eggs_first": true
}
},
{
"type": "TransferPokemon"
},
{
"type": "EvolvePokemon",
"config": {
"evolve_all": "none",
"first_evolve_by": "cp",
"evolve_above_cp": 500,
"evolve_above_iv": 0.8,
"logic": "or",
"evolve_speed": 20,
"use_lucky_egg": false
}
},
{
"type": "RecycleItems",
"config": {
"item_filter": {
"Pokeball":       { "keep" : 100 },
"Potion":         { "keep" : 10 },
"Super Potion":   { "keep" : 20 },
"Hyper Potion":   { "keep" : 30 },
"Revive":         { "keep" : 30 },
"Razz Berry":     { "keep" : 100 }
}
}
},
{
"type": "CatchVisiblePokemon"
},
{
"type": "CatchLuredPokemon"
},
{
"type": "SpinFort"
},
{
"type": "MoveToFort",
"config": {
"lure_attraction": true,
"lure_max_distance": 2000
}
},
{
"type": "FollowSpiral",
"config": {
"diameter": 4,
"step_size": 70
}
}
],
"map_object_cache_time": 5,
"forts": {
"avoid_circles": true,
"max_circle_size": 50
},
"websocket_server": false,
"walk": 4.16,
"action_wait_min": 1,
"action_wait_max": 4,
"debug": false,
"test": false,
"health_record": true,
"location_cache": true,
"distance_unit": "km",
"reconnecting_timeout": 15,
"evolve_captured": "NONE",
"catch_randomize_reticle_factor": 1.0,
"catch_randomize_spin_factor": 1.0,
"catch": {
"any": {"catch_above_cp": 0, "catch_above_iv": 0, "logic": "or"},
"// Example of always catching Rattata:": {},
"// Rattata": { "always_catch" : true }
},
"release": {
"any": {"release_below_cp": 0, "release_below_iv": 0, "logic": "or"},
"// Example of always releasing Rattata:": {},
"// Rattata": {"always_release": true},
"// Example of keeping 3 stronger (based on CP) Pidgey:": {},
"// Pidgey": {"keep_best_cp": 3},
"// Example of keeping 2 stronger (based on IV) Zubat:": {},
"// Zubat": {"keep_best_iv": 2},
"// Also, it is working with any": {},
"// any": {"keep_best_iv": 3},
"// Example of keeping the 2 strongest (based on CP) and 3 best (based on IV) Zubat:": {},
"// Zubat": {"keep_best_cp": 2, "keep_best_iv": 3}
},
"vips" : {
"Any pokemon put here directly force to use Berry & Best Ball to capture, to secure the capture rate!": {},
"any": {"catch_above_cp": 1200, "catch_above_iv": 0.9, "logic": "or" },
"Lapras": {},
"Moltres": {},
"Zapdos": {},
"Articuno": {},
    "// S-Tier pokemons (if pokemon can be evolved into tier, list the representative)": {},
    "Mewtwo": {},
    "Dragonite": {},
    "Snorlax": {},
    "// Mew evolves to Mewtwo": {},
    "Mew": {},
    "Arcanine": {},
    "Vaporeon": {},
    "Gyarados": {},
    "Exeggutor": {},
    "Muk": {},
    "Weezing": {},
    "Flareon": {}

}

}

Same here but not last version (Aug 8, 351ea76b6)

My password was longer than 15 characters which is what caused my issue

@leoxnidas
Hello
Sorry for my English, I am not English I use google trad
I had the same problem, you have a PTC account?
Because the PTC account need to validate the email address maximum two days after registration. You received an email with a link to confirm your email, if two days are passing you can log on with pokemonGo application and request the shipment of a new confirmation email. After that the error goes ;-)

no, i normally use google account, but i did try both, google and ptc config, and always get error

Not a bot bug, account related


GetODataV4Url from "Web Service Aggregate" not accessable

Is your feature request related to a problem? Please describe.
Hi everyone,
i just noticed that i cannot use function "GetODataV4Url" of table "Web Service Aggregate" .
Same for "GetODataUrl". "GetSOAPUrl" is accessable.
So you can't get any WebService URL without bothering the user.
Describe the solution you'd like
Is it possible to make this function reachable?
Best Regards,
Hendrik

Hi @hendrik244,
These functions were made external and will be available in the April release :)


Add a grabber for www.franceculture.fr.php

Hello,
This rule allows picoFeed to grab the Play button along with the content.

:+1:


Issue with Notion docs

    my codes

this not support

'''
    my code

'''
not support

  python     my codes

*  python codes *

Hi! Thanks for opening this issue. Yes, you are correct — Notion does not support syntax highlighting. I checked the Notion page and this is already correctly listed:

Syntax highlighting is not supported.


Can't always find a country to generate a locale

We get quite a few of our users marked as having a locale of just "en" in Android because it appears that locale.getCountry() is not a sufficient solution for all countries
I'm wondering if something more comprehensive like this solution might be considered an acceptable PR?
PS: Thanks so much for making this wonderful library, it has been so useful to us!

@samueljseay The language (and country variation) of the device is not really the same as the country where the user is currently.
An example with me: My phone is configured with en-US, but I live in France, and I don't want en-FR translation. Check solutions like https://github.com/pocesar/react-native-sim-data :)

Thanks @zoontek I'll look in to it. Appreciate the advice!


Make project code coverage a required check

I was wondering how people feel about making project-level code coverage targets a required check.  The goal is to either maintain or increase code coverage of unit tests over time.

That sounds fine, given good criteria. I find that I sometimes get a down rating when I move code from functions.js / buildHTML.js / buildMathML.js to the /functions/ subdirectory. That seems wrong.
So the target must be global, not on each file.

Definitely the overall goal is a good one. We can try the automatic requirement. If it gets annoying, an alternative would be human enforcement (based on the codecov reports we're already getting).


Instructions do not work.

poetry run python main.py
C:\Users\xxxx\Deep\blockagi>poetry run python main.py
Usage: main.py [OPTIONS]
Try 'main.py --help' for help.
Error: Missing option '--host'.
C:\Users\xxxx\Deep\blockagi>poetry run python main.py --host
Error: Option '--host' requires an argument.
C:\Users\vdrut\Deep\blockagi>

@bbecausereasonss Thank you for the bug report. We haven't tested much on Windows. I'll test and try to fix it tomorrow.
In the mean time if you can share you environment setup (Windows version, Python version, etc) that'd be super helpful.

Python 11, Winx64, fresh Poetry install as per instructions on their website.
See more (Its asking for a lot of missing flags) ->
C:\Users\vdrut\Deep\blockagi>poetry run python main.py
Usage: main.py [OPTIONS]
Try 'main.py --help' for help.
Error: Missing option '--host'.
C:\Users\xxxx\Deep\blockagi>poetry run python main.py --host <IP_ADDRESS>
Usage: main.py [OPTIONS]
Try 'main.py --help' for help.
Error: Missing option '--port'.
C:\Users\xxxx\Deep\blockagi>poetry run python main.py --host <IP_ADDRESS> --port 9898
Usage: main.py [OPTIONS]
Try 'main.py --help' for help.
Error: Missing option '--agent-role'.
C:\Users\xxxx\Deep\blockagi>poetry run python main.py --host <IP_ADDRESS> --port 9898

Seem like those might be missing ENV variables.
Have you tried copying/renaming the .env.example into .env file? This would load the variables from the file to be used with the script.
If you've already do that then I think the problem would most likely be with .env file and how it's loaded on Windows. Please confirm and I'd be able to check further.
Thank you!

Yep, you're right I missed that step now it's working!

Error on Windows..
(test) PS D:\test\BlockAGI> poetry install
Installing dependencies from lock file

Package operations: 62 installs, 3 updates, 0 removals

  • Installing pyasn1 (0.5.0)
  • Installing sniffio (1.3.0)
  • Installing anyio (3.7.0)
  • Installing cachetools (5.3.1)
  • Updating charset-normalizer (3.2.0 -> 3.1.0)
  • Installing frozenlist (1.3.3)
  • Installing h11 (0.14.0)
  • Installing hpack (4.0.0)
  • Installing hyperframe (6.0.1)
  • Installing marshmallow (3.19.0)
  • Installing multidict (6.0.4)
  • Installing mypy-extensions (1.0.0)
  • Installing protobuf (4.23.3)
  • Installing pyasn1-modules (0.3.0)
  • Installing pyparsing (3.1.0)
  • Installing rsa (4.9)
  • Installing typing-extensions (4.6.3)
  • Updating urllib3 (1.26.16 -> 2.0.3)

  CalledProcessError

  Command '['D:\\test\\Scripts\\python.exe', '-m', 'pip', 'uninstall', 'charset-normalizer', '-y']' returned non-zero exit status 2.

  at ~\python\Lib\subprocess.py:571 in run
       567│             # We don't call process.wait() as .__exit__ does that for us.
       568│             raise
       569│         retcode = process.poll()
       570│         if check and retcode:
    →  571│             raise CalledProcessError(retcode, process.args,
       572│                                      output=stdout, stderr=stderr)
       573│     return CompletedProcess(process.args, retcode, stdout, stderr)
       574│
       575│

The following error occurred when trying to handle this error:

  EnvCommandError

  Command ['D:\\test\\Scripts\\python.exe', '-m', 'pip', 'uninstall', 'charset-normalizer', '-y'] errored with the following return code 2

  Output:
  Found existing installation: charset-normalizer 3.2.0
  Uninstalling charset-normalizer-3.2.0:
    Successfully uninstalled charset-normalizer-3.2.0
  ERROR: Exception:
  Traceback (most recent call last):
    File "D:\test\Lib\site-packages\pip\_internal\cli\base_command.py", line 169, in exc_logging_wrapper
      status = run_func(*args)
               ^^^^^^^^^^^^^^^
    File "D:\test\Lib\site-packages\pip\_internal\commands\uninstall.py", line 110, in run
      uninstall_pathset.commit()
    File "D:\test\Lib\site-packages\pip\_internal\req\req_uninstall.py", line 432, in commit
      self._moved_paths.commit()
    File "D:\test\Lib\site-packages\pip\_internal\req\req_uninstall.py", line 278, in commit
      save_dir.cleanup()
    File "D:\test\Lib\site-packages\pip\_internal\utils\temp_dir.py", line 173, in cleanup
      rmtree(self._path)
    File "D:\test\Lib\site-packages\pip\_vendor\tenacity\__init__.py", line 291, in wrapped_f
      return self(f, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^
    File "D:\test\Lib\site-packages\pip\_vendor\tenacity\__init__.py", line 381, in __call__
      do = self.iter(retry_state=retry_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "D:\test\Lib\site-packages\pip\_vendor\tenacity\__init__.py", line 327, in iter
      raise retry_exc.reraise()
            ^^^^^^^^^^^^^^^^^^^
    File "D:\test\Lib\site-packages\pip\_vendor\tenacity\__init__.py", line 160, in reraise
      raise self.last_attempt.result()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\Administrator\python\Lib\concurrent\futures\_base.py", line 449, in result
      return self.__get_result()
             ^^^^^^^^^^^^^^^^^^^
    File "C:\Users\Administrator\python\Lib\concurrent\futures\_base.py", line 401, in __get_result
      raise self._exception
    File "D:\test\Lib\site-packages\pip\_vendor\tenacity\__init__.py", line 384, in __call__
      result = fn(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^
    File "D:\test\Lib\site-packages\pip\_internal\utils\misc.py", line 130, in rmtree
      shutil.rmtree(dir, ignore_errors=ignore_errors, onerror=rmtree_errorhandler)
    File "C:\Users\Administrator\python\Lib\shutil.py", line 759, in rmtree
      return _rmtree_unsafe(path, onerror)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\Administrator\python\Lib\shutil.py", line 622, in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
    File "C:\Users\Administrator\python\Lib\shutil.py", line 620, in _rmtree_unsafe
      os.unlink(fullname)
  PermissionError: [WinError 5] Access is denied: 'D:\\test\\Lib\\site-packages\\~harset_normalizer\\md.cp311-win_amd64.pyd'

  at D:\test\Lib\site-packages\poetry\utils\env.py:1525 in _run
      1521│                 output = ""
      1522│             else:
      1523│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
      1524│         except CalledProcessError as e:
    → 1525│             raise EnvCommandError(e, input=input_)
      1526│
      1527│         return decode(output)
      1528│
      1529│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:


8.0.0 -> 8.1.0 breaks  with TypeError when using Rollup

I'm not sure why upgrading to 8.1.0 breaks things. Or even why chalk would get included.
ansiStyles.color.closeRe = new RegExp(escapeStringRegexp(ansiStyles.color.close), 'g');
                                                                          ^

TypeError: Cannot read properties of undefined (reading 'close')

cspell-lib link:../../../packages/cspell-lib
└─┬ cosmiconfig 8.1.0
  └─┬ parse-json 5.2.0
    └─┬ @babel/code-frame 7.18.6
      └─┬ @babel/highlight 7.18.6
        └── chalk 2.4.2
cspell-lib link:../../../packages/cspell-lib
└─┬ cosmiconfig 8.1.0
  └─┬ parse-json 5.2.0
    └─┬ @babel/code-frame 7.18.6
      └─┬ @babel/highlight 7.18.6
        └─┬ chalk 2.4.2
          └── ansi-styles 3.2.1

cspell-lib link:../../../packages/cspell-lib
└─┬ cosmiconfig 8.0.0
  └─┬ parse-json 5.2.0
    └─┬ @babel/code-frame 7.18.6
      └─┬ @babel/highlight 7.18.6
        └── chalk 2.4.2
cspell-lib link:../../../packages/cspell-lib
└─┬ cosmiconfig 8.0.0
  └─┬ parse-json 5.2.0
    └─┬ @babel/code-frame 7.18.6
      └─┬ @babel/highlight 7.18.6
        └─┬ chalk 2.4.2
          └── ansi-styles 3.2.1

That's weird, there were no changes to any of the dependencies in package.json in 8.1.0.
Can you provide a full reproduction case (optimally just as a github repo I can easily clone and run to see the error) for this?

I tracked this down to a change in @rollup/plugin-commonjs. Version 23.0.4 works, everything past it does not. It seems to be due to this PR.
Please open an issue in rollup's repository.
If they refuse to fix this for some reason, the package having to adapt would probably be chalk or ansi-styles, not cosmiconfig.


Recover password does not have the right URL

This is a problem with the email, it's not adding the correct URL to recover the password

@Rub21, the domain name is taken from the server_url in settings.local.yml and is used in any links found in that email template. Could this be related to the DNS changes discussed in Slack during the deployment, i.e., was there previously an alias for web.openhistoricalmap.org that's no longer defined?

The issue has been fixed

@erictheise  the message of recovery email it still has OSM url, not sure where do we need to update that


include/core/SkSurface.h file not found

Description
Cannot use in my app. Also cannot even run the examples.

Clone the repo as instructed in the README (including making sure submodules are initialized)

I Add the .xcodeproj file to my app. (or even just open the rive-ios workspace)

Xcode fails to compile with the error "include/core/SkSurface.h" file not found in the file rive_renderer_view.mm

In an attempt to run the examples I run /make_dependencies.sh from the rive-ios directory

when that is completed, I attempt to run Schema RiveExample

The build fails with 63 errors. all of them some type of Undefined symbol

Provide a Repro
https://github.com/rive-app/rive-ios
NOTE: I am new to the XCode world, so I very easily could be overlooking something simple.

for more information,  here is another chunk of the error.

Hey @AirborneEagle - thanks for reporting this. The docs are definitely a little tough to follow for local development and running the example app - we'll get this cleaned up and the ios runtime in a better shape, docs-wise and example-wise.
You may need to make sure your submodules/rive-cpp submodule is pulled down. You could do that via git submodule update --init --recursive at the top level. Then you could run the dependency script (now it's sh ./scripts/build_dependencies.sh from the top level)

I had already initialized the sub repos, 😞
I am not sure what will solve this issue, but with the newly added #111  Swift Package Manager support, I no longer need it as a submodule 😄


Error occurred while enabling Gringotts v2.10.1-SNAPSHOT (Is it up to date?)

This error occurred in the following versions of Spigot and PaperSpigot (Paper):
Paper-1.13.2-b403.jar
spigot-1.13.2-R0.1-SNAPSHOT-b1966.jar
URL:
http://pasted.co/f943dd09

What Vault version are you using?

I'm using Vault 1.7.1 build 10 (https://ci.md-5.net/job/Vault/).

For all those who use the 2.10.1-SNAPSHOT version of Gringotts and want to run it on a server 1.13.x I recommend using the version of Vault 1.6.7 build 6 (https://ci.md-5.net/job/Vault/6/) to be compatible.
It works perfect in the 1.13.2 server version!

The only problem with that combination for 1.13.x are the items that don't detect them. 😞

I've got this running fine under 1.13.2 with Vault 1.7. It sounds like the version you're running is not supporting Vault 1.7 (Its looking for items, which was dropped from Vault 1.7).
Can you try my latest compile just in case you're running a slightly older version (I'm not 100% sure if the dev updates the version number as its a snapshot). https://github.com/bundabrg/Gringotts/releases
Run at your own risk. Otherwise check out the existing code and compile it to get the same binary.

It works great in version 1.13! I would recommend updating the resource for the Spigot page by this. :smile:

says the issue was on line 121. In the older version this line dealt with Vault Items and was removed by this PR: #182

Yes, you're right. I had put vault 1.6.7 and then 1.6.10 because the latest plugin update in Spigot does not accept vault 1.7 at 1.13.
I'm using Paper-1.13.2-b455 and this works properly! (Your update).

PS: Thank you very much for your help!

I think the slightly confusing thing is that the version on spigotmc is the same as the current build, but the one of spigotmc is the older version. Hopefully the dev see's this and uploads the correct version. For now I'll keep it maintained and push up PR's as needed.
Would you mind closing this issue if its working.

Exactly! The only problem is that the API version 1.13 is not implemented in this last update.
There is no problem with closing this issue, thank you very much again! :smile:


[SAGE-727] deleted .map files to support Django 4.1

https://eatclub.atlassian.net/browse/SAGE-727

I deployed this to django415 already via a custom branch and can confirm it resolved the issue 👍


Improve small road highway tag guessing

For KLASS 8 and 9 these are usually better mapped to track in rural areas, and service in semi-urban areas. There's not enough information in NVDB tagging to now which is which though.
Maybe by looking at things like road density, distance to urban areas etc the script could make better guesses. Fully correct will not be possible, but the goal is to minimize the need for manual changes.

I tried to experiment further with combinations of attributes. I have not found a specific attribute which denotes that a highway is used for forestry, but "Tillgänglighetsklass" is supposed to be registered on those highways, according to instructions. If you select highways according to the following criteria, I think you get 90% of what you want for track:

Functional class 9 (class 8 are important highways in urban/mixed municipalities)
"Tillgänglighetsklass" > 2 (class 1 and 2 are highway quality roads)
No street name (excludes a few semi-urban highways)
Not already tagged as Unclassified

The rest to be tagged as Service.
This seems to work across rural, urban and mixed municipalities. It is not perfect. For example you get "islands" of service at the end of tracks, but those are easily discovered during import.

Tillgänglighetsklass seems rather quirky in rural areas, points out very strange roads as 1 and 2 in Lycksele for example.
Anyway, I'm also working on this, which includes not only analyzing tags but geometry etc. It will be a few more days probably depending on time I get to spend on it, but I'll report back and see what I got. The goal is to get as many as possible correct over the whole spectrum rural->urban, but noting that 100% is not possible, so not trying to do something overly complicated as it will probably not perform better than a simpler model.

I have now added (a rather large) patch to "guess" upgrade track to unclassified and service, plus fix a quirk where short unclassified stumps are upgraded to residential.
This patch has most effect on rural areas. On outer areas of Göteborg and other semi-urban areas the change to look at if gatunamn is used (instead of övrigt vägnamn) and make those service instead of tracks took care of most roads.
On rural areas however the KLASS 8 level is highly unreliable, as well as tillgänglighetsklass, so the patch looks at properties of the geometry and makes upgrades from there. Roads that form connecting links become unclassified, then short stumps grouped together are identified as likely driveways, and if those are connected to a larger road it's upgraded to service. In a second pass other likely driveways nearby are identified and upgraded, and then in a final pass short tracks that links driveways to a larger road is upgraded to service. Log output from a run on Lycksele below. The result is not 100% perfect, but reduces the manual work significantly. The algorithm could be further refined, but I think this is a good start.
2021-03-19 22:08:45,739 - process - INFO - Upgrading short unclassified stumps to residential (if connected to residential)...
2021-03-19 22:08:45,782 - process - INFO - done (46 segments upgraded from unclassified to residential)
2021-03-19 22:08:45,783 - process - INFO - Making best guesses of upgrading highway=track to highway=unclasified or service...
2021-03-19 22:08:46,015 - process - INFO -   5134 tracks to be considered for upgrading (skipped 325 as they had gates, 896 as typical track names)
2021-03-19 22:08:46,113 - process - INFO -   69 tracks upgraded to unclassified as they form connecting links
2021-03-19 22:08:47,078 - process - INFO -   391 tracks upgraded to service as likely driveways connected to larger roads
2021-03-19 22:08:47,156 - process - INFO -   69 tracks upgraded to service as likely driveways near others
2021-03-19 22:08:47,341 - process - INFO -   42 tracks upgraded to service they form short links between roads and driveways
2021-03-19 22:08:47,342 - process - INFO - done (1312 of 5134 tracks upgraded to service)

cool. is that already on the main branch?

Yes, during this development phase everything goes in on the main branch :-). It's probably ok to run a new batch run now, but there's been quite many changes and not a whole lot of testing so we may see failures...

running the batch now

Very interesting approach, and a better result. A bit refining is needed as there are still "islands" of unclassified and service.
A fair number of rural streets with homes are still getting highway=track. Would it make sense to first restrict highway=track to roads were the "Tillgänglighetsklass" attribute is present (disregarding the value, which is not consistent)? In that case only a few segments at the end of the road lacks any "Tillgänglighetsklass" and needs to be added to highway=track.
I looked mostly at Uppsala and Falun, which are mixed-use cases.

The problem with these type of algorithms is that there is just no information except the shape of the geometry, and if you tune it for one use case it will perform worse in another.
Note that I regard "track" wrongfully marked as "service" as worse than "service" wrongfully marked as "track", which leads to that I have excluded some aspects that will make service grasp over too much. I'll do a check on tillgänglighetsklass layer again.

Looked at Tillgänglighetsklass, it cannot be used as a exclusive cutoff, but it could be used in a weighting algorithm.
Tillgänglighetsklass is indeed missing on driveways, but also on shorter forestry roads and some longer. I think though that if looking if there are nearby "safe" driveways one can with quite good confidence upgrade shorter roads.
There are a few "impossible" cases, and that's single isolated dwellings with a single short (or longer) road, these are exactly the same as the many short forestry roads for roadside clearcuts. However, as soon as there are pairs closeby, one can quite safely assume it's driveways as it's very rare for forestry roads to make short stumps close together, as there's no need.

I ran a test on Åstorp which i small municipality in south of Sweden, quite good for making fast testing on mixed-use.
By converting tracks to service which lacks Tillgänglighetsklass but are less than 1000 meters long and have a service/residential road within 300 meters and going through all of the roads affected by this rule I got ~75% right, and ~25% wrong, and a few that should be unclassified or residential instead. Similar numbers in Lycksele, although I think it will be a higher number correct there. The problem is that the bad ones turn up in places you are less likely to look, ie away from urban areas in lesser mapped places. When working with places like Göteborg, Uppsala or Falun basically every road is already mapped in some shape or form. Often with not so good geometry and lacking in tags, but it's there and need manual merging.
That's why I think it's actually more important to get things right in less dense areas, than in more dense ones, and I think it's better to keep too many tracks than upgrade too many to service.
Not sure how much value it is spending a lot more time on this though, it takes a lot of time tuning and testing these algorithms and it's a delicate balance, I'll probably put this to the side for a while.
I have recently working with importing semi-urban areas, and it's a lot of work. I have started with a older version of the script so more highways are wrong type, but I can say it's not there most time is spent anyway. One thing is that there's a lot more driveways mapped in OSM than exist in NVDB, and all these need re-attaching. And there's a lot of alignment errors of buildings and farmlands and other natural areas that needs correcting to fit roads, and one cannot just bulk-move as the alignment error varies over the surface.

the batch completed and at least it looks like all osm files were generated correctly: https://nvdb-osm-map-data.s3.eu-north-1.amazonaws.com/index.html

I just checked for myself the area where I live and I think there is a problem that I just found:
https://www.openstreetmap.org/#map=19/59.36117/17.96452 the road "landsvägen" is supposed to be a "lokalgata stor" which I think should translate to a tertiary road? it's currently marked as "residential" and that's definitely not true :)

It's a tradeoff, making lokalgata stor to tertiary in big cities makes them really overflow with tertiary. While that might be considered okay if we would start from scrach, it's not how they are mapped currently. Below is the current translation rules for city roads (those that exist in the gatutyp layer).
As you see only "Huvudgata" is translated to tertiary, and only if Funktionell vägklass is 5. For other roads also vägklass 6 becomes tertiary.
I've tuned these translations a bit differently, but anyway you do it there will be quite much mismatch from how it's mapped in OSM currently. When there is a difference, I change to what's in OSM while merging, unless it's obviously wrong. When discussing this a while back on the Facebook group it was very clear that the traditionalists is not keen at all of the idea of using fixed translations from NVDB classes to primary/secondary/tertiary etc, but they want this to be a mapper judgement, so I don't think we can make a translation that all will accept. So I think we will have to live with having to adjust these types quite often.
        if gatutyp == "Övergripande länk":
            raise RuntimeError() # should already be handled
        if gatutyp == "Huvudgata":
            if klass <= 1:
                tags["highway"] = "trunk" # 0, 1
            elif klass <= 2:
                tags["highway"] = "primary" # 2
            elif klass <= 4:
                tags["highway"] = "secondary" # 3, 4
            elif klass <= 5:
                tags["highway"] = "tertiary" # 5
            else:
                tags["highway"] = "residential"
        elif gatutyp == "Lokalgata stor":
            tags["highway"] = "residential"
        elif gatutyp == "Lokalgata liten":
            tags["highway"] = "residential"
        elif gatutyp == "Kvartersväg":
            tags["highway"] = "service"
        elif gatutyp == "Parkeringsområdesväg":
            tags["highway"] = "service"
            tags["service"] = "parking_aisle"
        elif gatutyp == "Infartsväg/Utfartsväg":
            tags["highway"] = "service"
        elif gatutyp == "Leveransväg":
            tags["highway"] = "unclassified"
        elif gatutyp == "Småväg":
            tags["highway"] = "unclassified"
        else:
            raise RuntimeError("Unknown gatutyp %s" % gatutyp)

I understand the frustration :)  I made a few examples with Tilgänglighetsklass. It seems to be a sufficient universe from which to get track. Of course the service "islands" at the end of branches would have to be included. And I am sure your more sophisticated method could identify a few unclassified and service roads among the tracks.
I just think there is too much guessing going on now. It would be clearer to base tagging on what is actually contained in NVDB (for good and for worse, just like primary/secondary etc), and I have found no other attribute than Tilgänglighetsklass to denote forestry roads. Sure it will not be perfect. That is why we do a manual import.
My experience is that importing is equally useful in semi-urban areas because the existing mapping is often with a big offset and with missing maxspeed, street name etc. Downtown areas in cities, on the other hand, are often well mapped because more mappers live there.
You could of course also have a special tagging variant for import needs in the north of Sweden.

I agree that it's useful to map already mapped areas, but what you get there is preset highway types.
No-one should ever delete existing roads and replace them with imported. It's just plain wrong if we suggest that it would be a valid workflow, and if anyone is caught doing that they should not be allowed to do imports. The old should be modified to new, either manually or using replace geometry tool. So why is modify instead of delete relevant in this context? When you modify you get clear feedback to set your highway type to either keep the old or change to the script's best guess.
This is not just about service vs track. It's about service, track, residential and unclassified, in semi-urban places all these four types may have same type of NVDB tags. You can't just look at the script output and decide if you like that or not, you must also look at what is pre-existing in OSM, and make a fair judgement of what's the best call, not just pick script result because that's the easiest. When doing that, it's clear that semi-urban places require careful work.
That said I think that there's quite of lot of bad highway choices in OSM already, so I correct it a lot too.
One can try that discussion rely on a static translations with the traditionalists :-). They couldn't care less about what is in NVDB. If NVDB provides wrong information, they are not happy to accept that just because it's in NVDB. They just look at what we want to be in OSM, not looking at NVDB at all. And I agree with them. We should not change our ways to map just to match what can be done with NVDB. We should look at how these ways should be tagged in OSM and Swedish tradition, and only then see if NVDB can match that. It can't.
The purpose of a guessing algorithm is to minimize the need to manually change. If the guessing algorithm causes a larger need to change than using statically incorrect conversions from various unreliable NVDB tags, then I agree it's probably better using a static conversion.
I think a problem here is though that we don't agree about the degree of correctness of a static conversion, I think you believe it's more correct than I do.
Let's look at an example from Uppsala.
This is currently in OSM today:

This is static conversion output from your script (don't know if I have the latest):

This is the guesswork conversion from my script from Matthias latest batch run:

I want to make changes to all these, also what's pre-existing in OSM.
Let's look at Örkälla, southwest of Skyttorp. It's residential in OSM, service in your script and track in mine. Residential is best call, but there's no tagging in NVDB for that, that's why our scripts get stuck on track/service.
Look at the forestry roads to the east. Here I say that my script has even better output than what's currently in OSM, which has too many unclassified. You don't want have gated roads as unclassified unless you have local knowledge and knows that the gate is permanently open or there's some other special reason. When it comes to quarries these are often abandoned, and type of road depends on if it's in active use or not.
Again, track is not tractor track exclusively in Swedish mapping tradition, it's also less important unpaved roads, which we have a lot of.
On secondary vs tertiary my script matches OSM, and the reason for that is that I use FPV-klass rather than funktionell vägklass for numbered roads which better match what OSM has now (but still not always...)
On the service roads, my script has made few mis-upgrades, but obviously misses quite many, and the reason of that is because it becomes too hard to differ between what's most likely roads to houses and what's forestry/argiculture tracks. However, these are generally in clusters which fit in your edit view, and I think it's better to have issues there than on long-winding roads.
For this example I certainly prefer working with the output the guesswork output makes rather than static rules.
Below a zoomed in example from Gödåker, which is seen center left on the others, after I have manually pre-processed the highway types:

Some short ones needed upgrading to service, but there's also some split work, Fredrikslund starts as service and transitions into track, ending with a turning circle (which needs adding, no turning circles on small roads in NVDB)
Gödåker and Lilla Gödåker, service, unclassified or residential? I have made it unclassified as there are several houses reached and the road is not specifically to one house. One might argue this should be residential, and I'm not entirely sure on this, but my current strategy which seems to fit mapping tradition well is to use unclassified unless there's a certain level of density when it becomes residential. Likewise Husby is split between unclassified and service.

Haha. Amazing thanks for the input. I just picked something close to me at random and sure enough found the hardest problem ;)
I really enjoy all the in-depth explanations. thanks a lot!

It seems you have not looked at my examples (which by the way only was produced to show the extent of tracks, not a complete end product). And I also seem to have failed to explain my approach. But I give up now.

Oops, I did miss the examples link, sorry for that, looking into it!

Here's a screenshot from the Tillgänglighetsklass script run, again sorry for missing that:

On a related subject: what I think we will have to do if we want to formalize this in an import process "available to all on full scale" is to start in a different end: making a document describing how highway types should be mapped in Swedish conditions, and specifically focusing on residential/unclassified/service/track, and get that accepted by the Swedish OSM community.
This document should not mention NVDB at all. It's just OSM on OSM's own conditions. In theory the OSM wiki should already have all the answers, but Swedish conditions are a bit special and there's an existing mapping tradition to take into account etc, and there are lots of corner cases.
I've learnt that examples is best, the document would contain similar to the screenshots above with different type of conditions and which types to use when.
When we have that established, the next step is to see how well this maps to what NVDB can provide. With that document in place, we have a common reference and it's easier to argue for what's correct in an OSM context.
Then one can start working on tuning the script to make as good job as possible to match that document guidelines, and then make a import process which describes caveats where one need manual changes etc.
I think it's the wrong way to look at what NVDB can do, and then convince ourselves that it is the best way to use OSM highway types. Even if NVDB functional class data would be 100% correct and consistent, which it isn't especially in the 7-9 range, highway types is not really defined in the same way as NVDB uses functional classes.
Sure for the range trunk to tertiary, one could argue for a static conversion, because there functional class maps to the highway types pretty 1 to 1, it's just that OSM mappers have in many places made a bit different decisions than NVDB, and NVDB itself has slightly different functional classes in different layers on these roads (depending on traffic type etc), so even there I think that manual adjustments are to be expected.
However, when it comes to residential, unclassified, service and track it becomes more complicated, it's sort of both functional class and use based. NVDB has no information if a road is used as a driveway. It indeed has information about residential, but much more area-based to maxspeed=50 areas and it misses quite a lot of villages that traditionally get residential in OSM. Actually I think I may be under-using residential myself currently, it's one thing I'll take up with the community. Service is about accessing something. And then we have track which is used a bit special in Swedish conditions, as we have this huge forestry network.
Making such a document will be lot of work though, and many discussions of people of differing opinions. I'm not totally against the idea of making such a document myself at some point, but it's probably not going to happen this year.
For my own purpose, the script is mature enough to entertain me with import work for years. As long as I am the single user that uses it on scale, I have lots of eyes on me, and informal explanations and discussions of what I'm doing is enough.

Here's a test with both tillgänglighetsklass and guesswork track upgrade active:
www.torger.org/anders/downloads/uppsala-new.osm
Note that I haven't run it with all layers so there's info missing, but most related to highway tag is there.

So I have added --small_road_resolve where you can specify prefer_service, prefer_track, or its static variants. Without static, it will on top run the guesswork upgrading of remaining tracks to unclassified/service (ie after static rules have been applied)
This is "prefer_service_static", should be most similar to NKAs, download here: prefer_service_static

This is "prefer_service", ie some track upgrades applied, not big difference in this view, but one can see some tracks have become unclassified,
prefer_service

And finally this is "prefer_track", ie same as the latest batch run
prefer_track

Just let me know which one you prefer as default setting, or if you want something else.
In the end I'm 100% sure that we will not get acceptance to use NVDB-derived highway tags, guessed or static, straight off as-is. Manual inspection and adjustment will be an absolute requirement on the importer. Just as using modify instead of delete followed by insert is an absolute requirement.
And then the task for the script is just about making an output that the importer finds effective to work with, and there we will see different preferences I suppose.
My preference is as follows: for roads that need tag adjustment I prefer when they are clustered (for example too many tracks inside a village is easier to correct than too many service spread out in the forestry network), and if I do mistakes I prefer to leave one too many with track tag than one too many service or unclassified tag. The reason for that is I think it's worse indicating that a road leads to something (service) when it doesn't than the other way around. On track vs unclassified I think when mistakes occur it's better with under-classing than over-classing.
By having a few different rendering options maybe we can cover most preferences.
What should the monthly batch run use though? Maybe two of the most popular. One reason to have two is to make it really clear for those that don't read docs that highway tag isn't set in stone.

I wonder if it would be an option to leave the highway tag out completely for the most "disputed" tags? That way they have to be set manually? Maybe that's too much work when actually importing the file...
Another option would be to set the flag by "kommun". So e.g. we know that setting X works best in Stockholm but setting Y works best in rural areas.
By the way, I have tinkered a bit further and now created a pipeline that produces vector tiles.
Have a look here: https://nvdb-osm-map-data.s3.eu-north-1.amazonaws.com/map.html#7.8/64.489/18.735 Currently on this map you see the differences in the tag highway. Additionally, when you zoom in a bit further, you get an overlay of the geometry that the nvdb2osm script outputs with the highway tag on the line. Maybe that helps.
My plan is to have this map toggleable so you can toggle if you want to see where the most differences are in lanes, maxspeed, highway etc.
But maybe I'll leave the highway tag out completely if it does maybe more harm than good.

I've also thought about leaving out the highway tag from script output. When working with previously mapped areas it's in theory quite practical actually, as you then never need to resolve highway tag conflict, with replace geometry you will automatically use what was previously there. However, it becomes a little bit messier to work with visually in the editor as the geometry then just becomes gray lines. It's also extremely easy for the user to remove highway tag from the layer if he or she wants, just select and delete.
If we are afraid that people are going to use the pre-made batch run data carelessly one could indeed leave it out of those files just for that reason. However, I'm not too worried about that, I think it's more about how we lay out the documentation, most want to do a good job. If one is worried about careless highway tag override I would be much more worried about people just deleting everything in view and pasting in the new layer, or messing up route relations, or turn:lanes on the bigger roads etc. There's so many wrongs you can do in addition to highway tag which we can't do anything about, except policing.
In terms of visualization of your comparison map (really nice by the way, just tested the new version), I'm thinking that maybe just differing between cycleway/footway/path and then all other highways is enough, and perhaps not with text which tend to distract overviews, but instead with different line width or dashed vs solid or something. When scanning for differences to find something suitable place to fix one will not look for mismatching highway tags, but things like maxspeed and geometry offsets.

Yeah you're right, maybe we're overly worrying...
Good idea. I'll put more effort down into the strengths of NVDB which is geometry and speeds, lanes etc and build the diff map around that.
I guess it would be really nice if you and @NKAmapper could come to a conclusion around the tagging so that we then can prepare the proposal for the import (for which NKA has already made a nice wiki page)...
My primary intention is really just to help as many people to improve OSM rather quickly :)

With risk of sounding a bit negative again, I think it will be tough to get a lot of people to actually help do the import work at least on short notice, so I would not feel too much in a hurry of the import proposal. This script has been out for two months, I've talked about it and the way I have put it to use frequently on the OSM Sweden facebook group, and still it's only the three of us that's active here on github. I was a bit naive and thought that there would be a bit more interest. There's no lack of appreciation of the import work I've already done, it's not that, but it's quite clear that among the active mappers there's no hunger for using import scripts to make their job quicker.
Part of the reason is that many mappers feel that it's a bit too difficult and they are afraid of making mistakes. I'd say that it's actually not that difficult, but if you never have used JOSM before there's a bit of a threshold for sure.
I've thought about making a "youtube training video" with JOSM and the import script and merging techniques. Text documentation gets a bit dense, the readme file to this script is already pretty crazy. I think a video could be very good way to get new mappers take the step and join the work. I've done some lame but effective instructional videos before for other reasons, so it's something I could do, but probably not until after summer if so.
There are a handful of advanced mappers in the active Swedish community that have the know-how already of course. I know of a few that already use NVDB since quite a while, I got my initial pointers to making my script from them, but they have their own scripts and methods, and still after knowing about my considerably more advanced and detailed script (ie much richer tagging) they still use their own methods and haven't even tried this script or commented. The thing is that I think fewer than we think has a specific desire to make OSM a complete working end product for whole of Sweden, but rather has pretty local interests or just find like mapping as a relaxing activity where the speed of mapping or richness of tags is not really that interesting.
So even with a fancy import guide, if we don't do anything more than that to attract mappers I think there's a big risk that there will only me and NKA doing all the full-coverage importing... to me that wouldn't be a failure though, I'm happy that I now can map much faster than I did before, I enjoy the mapping activity more this way, but sure I can't realistically finish Sweden on my own. In time I can cover my local interests though.

FYI: I collect edit guidelines based on what I use myself and on discussions in the facebook group in the README file. This should probably be refined with image examples and moved to an appropriate guideline document on the OSM wiki and be peer reviewed at some point. I've just kept it there for now for my own convenience. Anyhow, under the anonymous heading "Övrigt att tänka på" there's guidelines of how to relate to the script's highway tags and other things.

@atorger something completely different: I just found this: https://nvdb-osm-map-data.s3.eu-north-1.amazonaws.com/map.html#12.08/59.96192/15.85696 Is your script running amok? 😆 seems like it generates quite some chaos...

On the topic of your longer message: I do understand that it will probably not be a huge group that does the import and that's fine. Maybe a youtube tutorial is actually a really good idea. Part of me programming the pipeline and the map is the hope that this would lower the barrier of entry for some people. (OK and because it's actually really fun to tinker with these things 😉)
But as I understand the OSM customs, we should have an import plan ready and OK-ed by the community, right?
I would like to help with this as much as possible but this is a completely new field to me. I'd like to have @NKAmapper's guide for this as much as possible as you have already successfully driven such projects.

@matthiasfeist that's Fagersta kommun which looks ok in JOSM if you download it from the batch run page. If something like that would happen in the script it would never complete, so there's something going on somewhere else in the pipeline.
Yes I think your map will not only lower the barrier, but also make more people interested.
Yes we should have an import plan, especially if more people starts getting involved. I realize that it may be a bit problematic or at least uncomfortable to publish your "lower barrier of entry" project without having a import plan in place. I guess I could help with some text, just point me where I should write :-).
I don't know how you roll these things, but I guess one can invite people from the international mapping community to help, afterall many of the most active mappers in Sweden are not Swedes, top list here: https://osmstats.neis-one.org/?item=countries&country=Sweden. If we do that I think it becomes even more important with a solid import plan.

:) yes it seems to be a bug in my script. Damn...

Closing this. If we need further refinements, new algorithms etc, open a new issue.


Adds KubeCTL to base image

Pull Request Template
📲 What
Adds KubeCTL to base image
🤔 Why
In order that AIR can run kubernetes commands
🛠 How
Updated base dockerfile
👀 Evidence
https://dev.azure.com/amido-dev/Amido-Stacks/_build/results?buildId=21320&view=logs&j=6580f874-b455-5a3f-5e16-f00049fad720&t=f0a68e43-58d4-5b29-b1bd-df08f10a6302&l=432
🕵 How to test
Notes for QA

Approved


Fixed print issue

Fixed print to display parentheses around exponents. Added appropriate test cases.

@aliang8 one last change before this can be merged – please revert changes to package.json.

@aliang8 for an easy way to revert files to a previous commit see http://stackoverflow.com/questions/215718/reset-or-revert-a-specific-file-to-a-specific-revision-using-git.

Looks good.  Thanks for fixing this.


Backend verifies check mate

Task
Contract verifies check mate, a position in which one player's king is in check and has no valid moves left.
Detailed Description

Implement checkForCheckmate contract function that checks if this condition is indeed the case

Contract sets winner
Contract generates GameEnded(gameId, winnerAddress) event

Associated story
Story: #53
Notes

Games shouldn't end automatically after a move. As discussed, there should be checkForCheckmate and checkForDraw functions that the clients call when they have reasonable suspicion that one of them is the case. If these contract functions determine that the game indeed ended, they will then emit GameEnded events and could then automatically transfer any betting funds (if we implement that).

When this function is called, the game state is also updated if the validation is successful


feat(schematics): tree schematic

This PR adds a new schematic for the Material Tree module.
See example here: https://material-tree.stackblitz.io

Nudge @tinayuangao to take a look

Nudge @tinayuangao


Feature/scenario simulator v2

add some words to dictionary

Anja Sheppard : proprietary noun in exsample catalog file from carla
appinfo : keyword in miscobject .xosc file
autosize : keyword in drwaio file.
Carlo : Monte Carlo
EndofRoadCondition : keyword in OpenSCENARIO
STRNE : EXPECT_STRNE in gtest framework.
tfar : keyword in embree raycasting kernel library.
tnear : keyword in embree raycasting kernel library.


seL4 Microkernel: Visualizations for Abstractions

I like to understand things by characterizing them. Each thing I do or create should have a story and a purpose. Same goes for my writings. To make the writing 'seL4 Microkernel' I'm planing to design icons, characters or any uniquely identifiable visual elements. Each abstraction (service) provided by the seL4 needs to have one.

Capabilities
Untyped
Address Spaces
Threads
Endpoints
Notifications
Interrupts
~Faults~

The visualizations should be simple yet unique. It's not an easy job, but I believe I can create something nice ❤

Done as of f3c4a7e! I have removed "faults" from the list and replaced it with "address spaces".


@conda-forge-admin, please add user @maxrjones

Additional comment:
No response

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added user @maxrjones in conda-forge/xbatcher-feedstock#2.


Can't retrieve channel list

I am using tvheadend 4.1
{"sw_version": "4.1","api_version": 15,"name": "Tvheadend","capabilities": ["caclient","tvadapters","satip_client","satip_server","imagecache","timeshift","trace"]}

If I click on all channels or select a bouquet I get the error message "Request failed! Please check channel settings and tvheadend status."
The log contains following:
2015-11-28 15:29:01,409 (7ff4757fa700) :  INFO (__init__:104) - JSON-Request: getServiceGrid
2015-11-28 15:29:01,410 (7ff4757fa700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/api/mpegts/service/grid?start=0&limit=999999'
2015-11-28 15:29:01,411 (7ff4757fa700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/api/mpegts/service/grid?start=0&limit=999999'
2015-11-28 15:29:01,412 (7ff4757fa700) :  INFO (__init__:132) - JSON-Request failed: exceptions must be old-style classes or derived from BaseException, not str
2015-11-28 15:29:01,412 (7ff4757fa700) :  INFO (__init__:151) - Failed to fetch DVB services!
2015-11-28 15:29:01,412 (7ff4757fa700) :  INFO (__init__:242) - Could not create channellist! Showing error.
2015-11-28 15:29:01,417 (7ff4757fa700) :  DEBUG (runtime:924) - Response: [200] MediaContainer, 437 bytes
```

Exactly the same issue here, using TVH 4.0.7 (stable release). Did also not work with the latest 4.1 build.
Recordings are working fine in Plex, but live TV is impossible due to that bug.
2015-11-29 17:49:57,505 (7f736c9cc700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/api/mpegts/service/grid?start=0&limit=999999'
2015-11-29 17:49:57,507 (7f736c9cc700) :  ERROR (networking:217) - Error opening URL 'http://<IP_ADDRESS>:9981/api/mpegts/service/grid?start=0&limit=999999'
2015-11-29 17:49:57,508 (7f736c9cc700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/api/mpegts/service/grid?start=0&limit=999999'
2015-11-29 17:49:57,509 (7f736c9cc700) :  ERROR (networking:217) - Error opening URL 'http://<IP_ADDRESS>:9981/api/mpegts/service/grid?start=0&limit=999999'
2015-11-29 17:49:57,509 (7f736c9cc700) :  INFO (init:132) - JSON-Request failed: exceptions must be old-style classes or derived from BaseException, not str
2015-11-29 17:49:57,509 (7f736c9cc700) :  INFO (init:151) - Failed to fetch DVB services!
2015-11-29 17:49:57,509 (7f736c9cc700) :  INFO (init:242) - Could not create channellist! Showing error.
2015-11-29 17:49:57,512 (7f736c9cc700) :  DEBUG (runtime:88) - Sending packed state data (113 bytes)
2015-11-29 17:49:57,512 (7f736c9cc700) :  DEBUG (runtime:924) - Response: [200] MediaContainer, 437 bytes
Entering the "failing" URL in a web browser works.

UPDATE @CSchulz
I've found the problem! You need to tag the user in TVH which you use in the plugin as an "admin"!
@realriot :
Not sure if this is mandatory, if not you may consider changing this in the next update to not need admin level. I'm a bit careful not passing admin level logins to other services ;-)

I vaguely remember needing to do this a long time ago and haven't taken a look at it closely. I think it is unfortunately necessary as I have seen similar requirements in the iOS client as well. It may be needed to interact with the API. If I get a chance I'll take a look but it might be simpler to list it as a necessary requirement in the README.

I'm having this problem despite having an admin user in TVH. Any guesses as to what else could be causing the issue? TVheadend 4.1-1171~gaa13922 and Plex  <IP_ADDRESS>

I have a similar issue caused by encoding problems. Maybe your list of channels have special characters?

getting similar issues..
tvheadend                                   4.1-1258~gb480c92                        amd64        Tvheadend
2015-12-30 14:15:23,157 (7ff31bfff700) :  INFO (init:104) - JSON-Request: getChannelGrid
2015-12-30 14:15:23,160 (7ff31bfff700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/api/channel/grid?start=0&limit=999999'
2015-12-30 14:15:23,166 (7ff31bfff700) :  INFO (init:134) - JSON-Request successfull!
2015-12-30 14:15:23,166 (7ff31bfff700) :  INFO (init:104) - JSON-Request: getEpgGrid
2015-12-30 14:15:23,169 (7ff31bfff700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/api/epg/events/grid?start=0&limit=2000'
2015-12-30 14:15:23,304 (7ff31bfff700) :  DEBUG (networking:161) - Fetching 'http://<IP_ADDRESS>:9981/api/epg/events/grid?start=0&limit=2000' from the HTTP cache
2015-12-30 14:15:23,307 (7ff31bfff700) :  INFO (init:132) - JSON-Request failed: exceptions must be old-style classes or derived from BaseException, not str
2015-12-30 14:15:23,308 (7ff31bfff700) :  INFO (init:104) - JSON-Request: getServiceGrid
2015-12-30 14:15:23,310 (7ff31bfff700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/api/mpegts/service/grid?start=0&limit=999999'
2015-12-30 14:15:23,319 (7ff31bfff700) :  INFO (init:134) - JSON-Request successfull!
2015-12-30 14:15:23,319 (7ff31bfff700) :  INFO (init:242) - Could not create channellist! Showing error.
2015-12-30 14:15:23,321 (7ff31bfff700) :  DEBUG (runtime:88) - Sending packed state data (113 bytes)
2015-12-30 14:15:23,321 (7ff31bfff700) :  DEBUG (runtime:924) - Response: [200] MediaContainer, 437 bytes

Hi,
I had exactly the same problem and it was because of a channel name that the plugin didn't like.
Try disabling a few channel in tvheadend until it works, then re-enable them one by one (or ten by ten) and you should find the channel that the plugin doesn't like.
Damien

Hi,
I had exactly the same problem and it was because of a channel name that the plugin didn't like.
Try disabling a few channel in tvheadend until it works, then re-enable them one by one (or ten by ten) and you should find the channel that the plugin doesn't like.
Damien

Do you remember how the name looked like? It would be easier for debugging and finding the root cause.

Yeah I know but saddly the name was pretty standard, something like Poland TV.
I even checked in notepad++ to see if there was any special character but didn't find anything.
What else can I do to help ?

Perhaps you can try to reproduce it and store the webservice result.

results.txt

There are the results from the web service that i think is causing the issue

The message

I added some additional debugging and got this:
2016-01-10 23:39:04,364 (7f5de3fff700) :  DEBUG (networking:166) - Requesting 'http://<IP_ADDRESS>:9981/tv/api/mpegts/service/grid?start=0&limit=999999'
2016-01-10 23:39:05,068 (7f5de3fff700) :  INFO (__init__:126) - (2103, 'Data of size 6777367 is greater than the maximum size 5242880')```

@realriot ?

I was able to get it running by adding a max_size=10000000 parameter to the JSON.ObjectFromURL call in def getTVHeadendJson(), based on this reply: http://forums.plex.tv/discussion/comment/364315/#Comment_364315
Not sure if there are other issues though, since I can't test properly at the moment.

I'll add this workaround.
I really really want everyone to trigger the plex guys to update/offer the latest dev documentation. They really don't support the community anymore.

It sounds more like an issue in the paging of the result

It's neither a paging issue or a workaround. The reply is simply too large for the Plex libraries upper size limit. The real question is why you're requesting all services instead of just the configured channels.
Anyway, I'm not using this anymore.

Hi.
How I fix this?
Regards.

For me the solution was to delete the channel that was causing the error.
Sent from my iPhone

On 21 Apr 2016, at 23:49, espiman<EMAIL_ADDRESS>wrote:
Hi.
How I fix this?
Regards.
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

Thanks damhau.
I have only enabled one channel and problem persist.
I have also tried to modify the line 124 in init.py file, but problem persist.
json_data = JSON.ObjectFromURL(encoding='utf-8', url=url, headers=headers, values=None, max_size= 10000000)
Thanks.


Proposal: Webrtc Extension

Overview
This is a proposal (and intent to implement) basic webrtc facilities as an MRTK extension. The goal is to enable audio, video, and data channels for use from managed code, without the need to write custom native plugins (however, do note that native plugins will be implemented under the hood for the caller, as they are necessary to bind to webrtc on a given target platform).
The goal at this time is to begin with support the following targets:

Win32 Standalone
Android
WinUWP (non-ARM at this time)

Support for other targets (with ARM being the likely next candidate) can be added in the future. The planned approach will mirror the webrtc unityplugin example, and will be largely a refactor from bengreenier/webrtc-unity-plugin which supports webrtc (m71) facilities and includes some examples.
The native plugin work (a slight modification from google source, to support WINUWP and il2cpp) is based on webrtc-uwp, and can be seen in this PR.
In my opinion it's worth including a disclaimer here that this is a basic, best-effort solution and is not intended for high-scale production use at this time, but is designed to get us closer to a production story. I'm thinking of it as one of the core pieces in a larger puzzle. Hopefully that makes sense to folks.

This sounds like a great initiative! Do we know whether MS has any plans in this area, seeing that they are already quite active developing the underlying WebRTC for UWP repository? Maybe it’s possible to coordinate our efforts? Either way, I’d be happy to contribute if I can, I’m currently building a similar library that works with HoloLens + iOS, however it is currently built to work with HTK, so some work would be needed there as well.

Hi @anders-lundgren - I actually work at Microsoft 😸 @jamescadd (who is driving the webrtc-uwp work) is super looped in on this work, as well as the MRTK Microsoft folks. As I mentioned we think this is likely to be one of the building blocks of a larger webrtc solution for MRTK folks (though we not fully committed to this approach at this point, to be transparent).
Happy to collaborate going forward, though my immediate goal is to submit a PR into a feature branch for the existing work I've done.

@anders-lundgren another benefit to this ending up as a MRTK extension is that it doesn't necessarily have to take dependencies on the core/foundational MRTK components. We should be able to deliver a WebRTC component that can be packaged/distributed independent of the rest of the MRTK, which hopefully will be compatible with your HTK project as well. :)

Assigning to Jerome, he will be driving webrtc work.

@anders-lundgren just to update you quickly I am currently working on extending the work @bengreenier did beyond its original scope, to provide a more polished library which comes with added features (ARM support being a notable one), which is why Ben's PR #3664 was closed in the meantime. This was a coordinated decision with him. I am also in contact with @jamescadd to leverage the work done by the WebRTC UWP team. Please bear with me for a little while until we finalize our plans to make our repository public. This will not be published inside the current MRTK-Unity repository though.
Do you have any example of interaction with the existing MRTK components you would be interested about? Or are you simply looking for an audio and video streaming solution in general, to use in an app which also makes use of MRTK? What do you mean when you say your library "works with HTK"? We do have plans to make any integration between the two as smooth as possible, though nothing concrete in practice yet. I'm open for suggestions!

@djee-ms I've been using the code from Ben's code request on and off for a bit now.
Couple things I've had to fix/adjust:

Data Channels were not setup correctly. Reliable needs to be false or change data transfer mode.
Added hooks to register Ice state changes and signal state changes. Otherwise I am not sure how you would tell if the other user dropped the call.

Couple things I've had to work around but haven't touched:

Order of operations is fickle wtih Data channels. The simple_peer_connection has a data_channel property separate from the peerconnection.cc's list of data channels.  Sometimes it checks the local data_channel, sometimes it checks the list and that can cause issues.  Example:  Peerconnection automatically creates data channels based on the RemoteDescription/offer but the simple_peer_connection does not. So you can easily get into a state where the data channel already exists and you can't add it to simple_peer_connection.
Setting audio mode (ismute, isRecord) twice with the same value will cause Unity to crash (at least in debug mode).  There is a check to see if the sink already exists and if so boom.

If this is at all helpful I can keep adding more notes as I find them.  I'm currently having issues with performance and having a delay on the video/audio on the device. I was trying to set a min/max framerate of 1 frame per second (we don't need fast video for our use case) but I haven't had much luck yet.
Our use case is the remote assist skype like use case for 1 on 1 help that we are adding natively to our app.

@markgrossnickle thanks for the info!

I have rewritten some of the data channel code to get support for adding/removing multiple channels, and not open any by default, so the existing issues should be fixed. I also exposed both the ordered and reliable options, as those can be of use depending on the use case one has for data channels. Also Ben's code uses SimplePeerConnection which has RTP data channels but support for this is being deprecated and removed by Google as I understand. The alternative, SCTP, allows both reliable and unreliable data channels. And I enabled DTLS/SRTP too.

I added some hook on signaling change to get notified when the connection has been established, so that e.g. the local media pipeline can start to render the video/audio feeds. I use the transition to stable for that. My interpretation is that you should avoid using the ICE/DTLS state, which tells you if internally a low-level connection was established at the network level, but not if the WebRTC connection itself has completed its establishing steps.

I didn't look at connection dropped yet, thanks for the reminder. I think that you should probably also use the closed signaling state for that. My intuition is that again the ICE/DTLS state is for internal network level connection (socket connected), but not the WebRTC peer connection itself. Though in the case of dropping it's probably almost the same.

Incidentally, note that currently the code in Ben's plugin and WebRTC UWP SDK is mixing the ICE and DTLS states, as explained in this bug. This has been fixed here, and now those are 2 separate states, but the change didn't reach WebRTC UWP SDK yet.

@djee-ms This is fantastic. Any chance I can check out the source code and help bang on it?

@markgrossnickle I wish, but unfortunately not yet. At the moment the repository I am working on is private. I am working to get it public as soon as possible, even if the code is not ready for production, at least to get some feedback. But the administrative part takes some time so please bear with me. I'll post here as soon as the source code is publicly available.

@djee-ms
Understood. Looking forward to when you can share.
Two last questions and I'll let you be ;)

Performance isn't great on the Hololens, at least on the branch I am working on.  Would it be possible to expose the frame rate of the video capture and/or playback?  My current use case just needs to show a frame about every second or so as not much will be changing.  I tried setting kMinFrameRate and kMaxFrameRate to both be 1 and pass those constraints to CreateVideoSource in the AddStream function of simple_peer_connection but it doesn't seem to do anything so perhaps I am in the wrong spot. I'm faily new to WebRTC.

Large file sharing. Perhaps this is tackled with changing to SCTP, but with the current setup the file size limit on the data channel seems low and I am unable to share a photo. I was debating fragmenting the photo and then putting it back together or looking for another way to share it such as uploading to our server and then passing the URL.  Do you have any plans for larger data sharing? I know the sharing demos seem to share anchors and other positioning data. Are anchors and everything else being demo'd pretty small or do you guys have a work around for larger files in the works?

@markgrossnickle
You can use FramesPerSecondChanged event on Conductor object. Event handler could be added in  ControlScript.cs script. Please check how it is handled in PeerCC XAML application. It could be used as a reference.
https://github.com/webrtc-uwp/PeerCC/blob/master/Client/ViewModels/MainViewModel.cs#L296

@morosev We don't have a conductor in the Unity build (at least the example I have access to).  But thanks for the link, this should point me in the right direction for implementing it in the Unity build!

@markgrossnickle You can find the framerate counter implementation in WebRTC UWP wrapper here: https://github.com/webrtc-uwp/webrtc-apis/blob/master/windows/wrapper/impl_webrtc_MediaStreamSource.cpp#L510

Right now I am evaluating the current capabilities of webRTC and would like to ask a few things about this PR:

does it have any reference to the Microsoft webRTC universal samples project?
does it have people actively working on it, and preferably a roadmap of upcoming supported features?
What's the current range of features? (video-, audio-streaming,... Data distribution (avatar movement))...

can someone elaborate?

Hi @Alexees,
To answer you questions:

Not that I know of. However those samples are authored by the same team which works on the WebRTC UWP SDK project, and the new library I am working on is taking a dependency on that project to provide support on UWP platforms like HoloLens 2.
Yes, I took over this work from @bengreenier and I am working on that project full time. I am currently working on a separate repository, and I intend to make it public as soon as possible, hopefully in the next few weeks, as I finalize some administration around that task. This will be an experimental preview version, so the API may still change, but I believe can be of help for people doing similar things to take example from. I do not have a roadmap yet, though I intend to get one eventually based on feedback.
Initial features will support audio and video streaming to a single remote peer, with hopefully Mixed Reality Capture working for HoloLens devices. I also added ARM (32-bit) support. The libraries will not be tied to Unity, so can be used in any C++ or C# application. Some Unity integration will be provided though. Data distribution as you call it will be supported in the sense that you can open one or more data channels per peer connection, but automatic state synchronization is out of scope of this project.

I am welcoming any suggestions about features you are interested in which are specific or closely related to WebRTC, like multi-peer support (MCU, SFU), access to specific parts of the underlying WebRTC API that have not been mentioned yet on this discussion thread, etc. However any technology built on top of WebRTC data channels is unfortunately out of scope for this project.

@djee-ms thanks for the thorough answer

This is in the works, please join the sharing weekly shiproom for the latest updates.

New repo:  https://github.com/Microsoft/MixedReality-WebRTC

New repo: https://github.com/Microsoft/MixedReality-WebRTC

The link seems to be broken

@Alexees Ah sorry, looks like I jumped the gun. I'll update here when the link works.

@alexees https://github.com/microsoft/MixedReality-WebRTC/ Good to go now :)


Management.ServiceFabric always retrieves empty Application/Service lists

Using the following code results in both appList and serviceList containing an empty list.
var client = new ServiceFabricManagementClient(tokenCreds) { SubscriptionId = subscriptionId };
var appList = client.Applications.List(resourceGroup, cluster.Name);
var serviceList = client.Services.List(resourceGroup, clusterName, appname);

The client uses {ResourceURI}?api-version=2017-07-01-preview to make requests. Which returns the following (empty) response:  { "value": [] }.
Overriding API version to 2016-09-01 (the one Azure Portal uses), correctly returns the list, but parsing fails (all properties in the parsed object are null).
Parsing fails because property names of the properties object in the returned JSON (below) are capitalized, but in the C# class definitions they start with the lowercase letter.
{
  "value": [
    {
      "id": "/subscriptions/{...}/resourcegroups/{...}/providers/Microsoft.ServiceFabric/clusters/{...}/Applications/GettingStartedApplication",
      "properties": {
        "Name": "fabric:/GettingStartedApplication",
        "TypeName": "GettingStartedApplicationType",
        "TypeVersion": "1.0.0",
        "Status": 1,
        "Parameters": [
          {
            "Key": "GuestExeBackendService_InstanceCount",
            "Value": "-1"
          },
          {
            "Key": "MyActorService_MinReplicaSetSize",
            "Value": "3"
          },
          {
            "Key": "MyActorService_PartitionCount",
            "Value": "9"
          },
          {
            "Key": "MyActorService_TargetReplicaSetSize",
            "Value": "3"
          },
          {
            "Key": "StatefulBackendService_MinReplicaSetSize",
            "Value": "3"
          },
          {
            "Key": "StatefulBackendService_PartitionCount",
            "Value": "6"
          },
          {
            "Key": "StatefulBackendService_TargetReplicaSetSize",
            "Value": "3"
          },
          {
            "Key": "StatelessBackendService_InstanceCount",
            "Value": "-1"
          },
          {
            "Key": "WebService_InstanceCount",
            "Value": "-1"
          }
        ],
        "HealthState": 1,
        "ApplicationDefinitionKind": 0,
        "Id": "GettingStartedApplication"
      }
    }
  ]
}

@val-janeiko
Were you able to resolve this and get apps/services ? I am stuck too..

@val-janeiko are you still having issues with querying? This issue has been resolved in the runtime.

@val-janeiko are you still having issues with querying? This issue has been resolved in the runtime.

@peterpogorski  Yes.
I just checked with the latest package from NuGet: v1.1.0. By default requests are made with api-version=2017-07-01-preview. The response for list of services contain a single service but .NET library fails to parse upgradeReplicaSetCheckTimeout property, returned one is a TimeSpan but it expects an Int. List of Services is still the same - empty.
I have also checked with api-version=2019-03-01. Results are the same as with 2017-07-01-preview.
Using api-version=2016-09-01 correctly returns both lists, but only the IDs of the resources are parsed in .NET objects. Everything else is set to null.

@val-janeiko thanks for reporting. We have made the fix for version 2019-03-01. We will publish the sdk with this new version shortly. I'll update this issue once it is done.

The latest SDK package Microsoft.Azure.Management.ServiceFabric 1.2.0  uses api-version 2019-03-01 wich fixes this problem.


Replace classification accessor on Job Posters with 1-n relationship

Resolves #1445.

@gggrant  The problem was the the previous field classification_code was holding the value of the string value (eg. AS). With the new changes you made, classification_id holds the id (int). However, the data on the react side is expecting the string value, so I had to refactor a few things.


Bank Tag Tab Monster Icons

If you tag your weapons for a specific monster, Eg. setup for a sara trip, or bandos, etc, then it doesnt really make sense to tag it with a picture of your weapon, since that weapon is used at multiple places. If we could tag it with a picture of the monster instead, like you get with a slayer assignment task.
Describe the solution you'd like
When you change icon, you can type in the name of the monster, and  if it matches it will use that as the icon instead of an item.

Duplicate https://github.com/runelite/runelite/issues/6021


consider gatsby-remark-copy-linked-files use

Should consider this plugin if intending to add downloadable files to blog posts remark-copy-linked-files

No plans to use this at this time. Will reopen if needed.


Future of maxCols property.

Should this be part of the one/many responsive solutions we have might have to Armstrong, rather than built-in?

With the inclusion of the responsive stuff, maxcols doesn't really bring anything to the table so I'll probably remove it unless anyone objects?

I don't use maxCols so don't mind it going. Does prompt a question about responsiveness in Armstrong but I'll create another issue for it.

Feature removed


Allow CharSquence in 'parse*()' method except for String

It is best practice and more flexible to expect the interface of String, CharSequence als input of the parser methods. Note that it will break the BinaryToTextEncoding interface (although this should be easy to fix)

Fixed in 04e18dd9031f4d884999d5175a01f3c116d4203b


Arguments to karate.call pollute namespace of subsequent calls

I think I found a regression from 0.9.2 (compared to 0.9.3.RC2) which was probably introduced by the fix in #722. It seems that (maybe only some) calls can pollute the namespace of the subsequent calls. In this case I found, the arguments passed to one karate.call are then also visible in any subsequent karate.call.
I have prepared a reproducing example:
scopebug.zip
Let me know if you need any more info!

took a look and am giving up for now. the report for this scopebug example looks OK to me with the above fix.

see if you can solve. hint, it is this line that appends the result (which is reported) onto the right "parent" https://github.com/intuit/karate/blob/develop/karate-core/src/main/java/com/intuit/karate/Script.java#L1763
maybe we can add a reportContext variable to the CallContext class which is already a little cluttered

Ah, I think I worded my previous reply misleadingly. Your change does fix the bug exhibited by the scopebug example, but regresses the example attached in #728.

@kkris okay can you see if it works fine now ?

Both #728 #722 and #795 are fixed with your change if I change the context as you suggested above:
callContext.reportContext.addCallResult(result) instead of callContext.context.addCallResult(result)

@kkris wow I totally missed the most important part, thanks for catching that ! guess this is fixed now

Yes, perfect!

released 0.9.3