Advanced Mapping: add field mappings cascade delete

Critical Changes
Changes

"Deleting" an object mapping will now "delete" all of its field mapping children.

Issues Closed
New Metadata
Deleted Metadata

**lurch: attach W-035505

Tracking W-035505

**lurch: attach W-035505


haskell.compiler: ghc921 -> ghc922

https://www.haskell.org/ghc/download_ghc_9_2_2.html

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@ofborg build haskell.compiler.ghc922

We won't be able to merge this into master sadly, as the ghc-lib-{,parsers,parsers-ex} for 9.2.1 stop building with 9.2.2. Updating the packages in question on master is probably more trouble than it's worth.

Why is merging this into master impossible because of that? Wouldn’t making it available as pkgs.haskell.compiler.ghc922 not break any existing packages?

The available package versions are the same for all 5 package sets, we merely pick and choose different ones via overlays for different compiler versions some of the time. For better or worse the entire package set(s) is updated as a unit. This creates the situation where updating GHC 9.2 causes regressions in the 9.2 package set (ghc-lib* stops working), but updating the package set would certainly cause regressions in other package sets. There would certainly have been a hack to work around this, but I chose saving some maintainer time over shipping GHC 9.2.2 as quickly as possible in this instance.
In hindsight, it would have been possible to expose an additional ghc922 package set in addition to an unchanged ghc921 package set, maybe I'll go down this path the next time around.
If you absolutely need GHC 9.2.2 for something, I would assume — though without testing it  — that the GHC 9.2.2 experience on haskell-updates is comparable to the 9.2.1 experience on master (albeit lacking the little binary cache master has for aarch64-darwin).

Also normally we wouldn‘t expect haskell-updates to go without a merge for so long. So the drawback of not doing stuff on master was higher than expected.


Docs say

File: engine/reference/commandline/dockerd.md, CC @mstanleyjones

I'm sorry, what problem did you see in the doc? There isn't much info here.

Sorry, I did not mean to open this. I created a much larger issue and I guess along the way I made this one by accident.


use extern "c++" for defining C++ code

At the moment, uv_tty_set_mode is defined after closing extern "C" opened at the beginning of uv.h, in the hope that it would be defined as a C++ function.  However there is no such guarantee, the line that included uv.h might be surrounded by extern "C".
This PR uses extern "C++" instead to define the C++-only function.
fixes #254

/cc @bnoordhuis

Sorry for the delay.  LGTM if it LGTY, @saghul.

No worries @bnoordhuis! Thanks a lot @kazuho, landed in 252f362

My pleasure! :+1:


Master-to-next merge

Automatically created merge PR, pulling recent master changes into next.

This pull request introduces 1 alert when merging 79e246f961f2f5712196380a80555e87a9d975b0 into 6b27dcabc567068dc9149d0aa8f74fee8cef633e - view on LGTM.com
new alerts:

1 for Redundant ToString() call

Comment posted by LGTM.com

Why is there no Language-Tests/JavaScript running?

This pull request introduces 1 alert when merging b4f400fb23f24c835658858899faba4c14943d06 into 6b27dcabc567068dc9149d0aa8f74fee8cef633e - view on LGTM.com
new alerts:

1 for Redundant ToString() call

Comment posted by LGTM.com

This pull request introduces 1 alert when merging 42cf76027afd8da9c6c9f88b6afa696e5a0c8059 into 6b27dcabc567068dc9149d0aa8f74fee8cef633e - view on LGTM.com
new alerts:

1 for Redundant ToString() call

Comment posted by LGTM.com


[Request] Staging/Stage recovery

What do you think is missing or what do you think can be done better
[Describe in detail what you want to see and how it makes the mod better]
I think you should add stage/booster separation because

It's more realistic
You could be more efficient with your launches
You could recover them SpaceX style.
I understand if it's too much of a hassle, and I don't blame you.
Thanks for your consideration.

Duplicate of #316


nixos/redis: enable vmOverCommit by defaul; don't disable transparent hugepages

Description of changes
See each commits. Tested on a nixos machine by booting with transparent_hugepage=madvise and checking the redis logs which don't mention anything anymore.
Closes #90602
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Just added a release note entry

Do we want to merge this before the 24.05 branch off?


Search for a Person NHS admin v2

As NHS admin I want to be able to search for a person using next possible filter parameters:

possibility 1:
tax_id

possibility 2:
phone_number

possibility 3:
first_name
last_name
birth_date

possibility 4:
national_id

and sort by  person_name (first_name&second_name), birth_date, tax_id, national_id
We need to implement WS according to spec

@warehousman @danakonovalenko please review list of combinations


Bugs/Miscompilations

Hi,
I am stresstesting Arbor's NMODL compiler and ran into some cases that I think are mishandled by nmodl
as well:
KINETIC under conditionals
NEURON { SUFFIX test_kinetic_alternating_reaction }

STATE { A B }

BREAKPOINT {
  SOLVE foobar METHOD sparse
}

KINETIC foobar {
  LOCAL x, y

  x = 23*v
  y = 42*v

  if (v<0) {
    ~ A <-> B (x, y)
  }
  else {
    ~ A <-> B (y, x)
  }
}

Arbor
Throws an error; which I think should be the expected behaviour
nmodl
After 5/local rename
NEURON {
    SUFFIX test_kinetic_conditional
}

STATE {
    A
    B
}

BREAKPOINT {
    SOLVE foobar METHOD sparse
}

DERIVATIVE foobar {
    LOCAL x, y
    x = 23*v
    y = 42*v
    IF (v<0) {
    } ELSE {
    }
    A' = (-1*(x*A-y*B))+(-1*(y*A-x*B))
    B' = (1*(x*A-y*B))+(1*(y*A-x*B))
}

which is likely not what a user expects.
Alternating between reaction and non-reaction statements
NEURON { SUFFIX test_kinetic_alternating_reaction }

STATE { A B C D }

BREAKPOINT { : NOTE we need a newline here :/
  SOLVE foobar METHOD sparse
}

KINETIC foobar {
  LOCAL x, y

  x = 23*v
  y = 42*v

  ~ A <-> B (x, y)

  x = sin(y)
  y = cos(x)

  ~ C <-> D (x, y)
}

Arbor
Produces the same code as nmodl does, see below. However, I think this should constitute
an error and Arbor will adopt this in the next version.
nmodl
After 5/local rename
NEURON {
    SUFFIX test_kinetic_alternating_reaction
}

STATE {
    A
    B
    C
    D
}

BREAKPOINT {
    SOLVE foobar METHOD sparse
}

DERIVATIVE foobar {
    LOCAL x, y
    x = 23*v
    y = 42*v
    x = sin(y)
    y = cos(x)
    A' = (-1*(x*A-y*B))
    B' = (1*(x*A-y*B))
    C' = (-1*(x*C-y*D))
    D' = (1*(x*C-y*D))
}

Again, I think this is unexpected and likely constitutes a user error.

thanks, @thorstenhater!
The #89 mentions conditional statements in the title but there is no specific example there. These are bugs and I vaguely remember discussion around these:

For 1, if NEURON does handle such MOD file correctly, we were wondering if that should be supported in NMODL as well
For 2, we were also saying it would be easy for user to rewrite mod file. Or, whether NMODL should do something smart.

Just tagging here @nrnhines to confirm if nocmodl from NEURON is handling above examples.

For 1. the definition of correct should be this, I assume
KINETIC foobar {
  LOCAL x1, y1, x2, y2

  x1 = 23*v
  y1 = 42*v
  x2 = sin(y1)
  y2 = cos(x1)

  ~ A <-> B (x1, y1)
  ~ C <-> D (x2, y2)
}

which is readily doable.
For 2.) I can imagine a similar transformation, but is much more involved, as one would need to hoist
sideeffects across the conditional. However, things might become infeasible ad/or confusing to the
user. Essentially -- I assume, since I see no better way -- one would compute all rates for all combinations
of the conditionals and set them before producing the ODE system. That however is subject to combinatorical
explosion, $2^N$ with $N$ conditionals.
Essentially this
KINETIC foobar {
   LOCAL x, y
   if (a()) {
     x = f()
     y = g()
     ~ A <-> B (x, y)
  } else {
     x = h()
     y = i()
     ~ 2B <-> C (y, x)
     ~ C <-> D (x, y)
  }
}

should become
KINETIC foobar {
   LOCAL rAB, iAB, r2BC, i2BC, rCD, iCD
   if (a()) {
    rAB = f()
    iAB = g()
    r2BC = 0
    i2BC = 0
    rCD = 0
    iCD = 0
  } else {
    rAB = 0
    iAB = 0
    r2BC = i()
    i2BC = h()
    rCD = h()
    iCD = i()
  }
  ~ A <-> B (rAB, iAB)
  ~ 2B <-> C (r2BC, i2BC)
  ~ C <-> D (rCD, iCD)
}

which already gets taxing.
Can we document the definitions of the correct outputs somewhere?

The kinetic schemes shown with conditional and alternating reaction/non-reaction work properly with NEURON's nocmodl (and are stable as long as the rates are positive).  I do believe, though, that an argument can be made that conditionals are a bad idea as there are implicit requirements  in regard to number of states being the same. To get sensible results with simple tests. I modified the examples to be.
$ cat test1.mod
NEURON { SUFFIX test_kinetic_conditional }

STATE { A B }

INITIAL {
  A = 1
  B = 0
}

BREAKPOINT {
  SOLVE foobar METHOD sparse
}

KINETIC foobar {
  LOCAL x, y

  x = 23*fabs(v)
  y = 42*fabs(v)

  if (v<0) {
    ~ A <-> B (x, y)
  }
  else {
    ~ A <-> B (y, x)
  }
}

and
$ cat test2.mod
NEURON { SUFFIX test_kinetic_alternating_reaction }

STATE { A B C D }

INITIAL {
  A = 1
  B = 0
  C = 1
  D = 0
}

BREAKPOINT { : NOTE we need a newline here :/
  SOLVE foobar METHOD sparse
}

KINETIC foobar {
  LOCAL x, y, z

  x = 1*fabs(v)
  y = 1

  ~ A <-> B (x, y)

  z = fabs(sin(y))
  y = fabs(cos(x))
  x = z

  ~ C <-> D (x, y)
}

Test1 with a square wave voltage gives

and Test2 with a step function voltage gives

Alternating Reaction Equations & Assignment
NMODL has changed how it handles KINETIC blocks. All rates are converted to LOCAL variables. Therefore, the above example turns into the following:
KINETIC foobar {
    LOCAL x, y, kf0_, kb0_, kf1_, kb1_
    x = 23*v
    y = 42*v
    kf0_ = x
    kb0_ = y
    ~ A <-> B (kf0_, kb0_)

    x = sin(y)
    y = cos(x)
    kf1_ = x
    kb1_ = y
    ~ A <-> B (kf1_, kb1_)
}

which you'll note that it can be safely reordered:
KINETIC foobar {
    LOCAL x, y, kf0_, kb0_, kf1_, kb1_
    x = 23*v
    y = 42*v
    kf0_ = x
    kb0_ = y

    x = sin(y)
    y = cos(x)
    kf1_ = x
    kb1_ = y

    ~ A <-> B (kf0_, kb0_)
    ~ A <-> B (kf1_, kb1_)
}

and from there converted to an ODE.
This was implemented to make MOD files like work:
https://modeldb.science/98005?tab=2&file=D2modulation/ca_ch.mod
which contains:
	FROM i=0 TO NANN-1 {
		dsqvol = dsq*vol[i]
		~ ca[i] + Buffer[i] <-> CaBuffer[i] (k1buf*dsqvol,k2buf*dsqvol)
	}

which after loop unrolling turns into the exact issue reported.
It seems reasonably clear that the intended behaviour is:
		~ ca[i] + Buffer[i] <-> CaBuffer[i] (k1buf*dsq*vol[i],k2buf*dsq*vol[i])

and expressing it via dsqvol is an "optimization". Similarly, with the reported example, there's a sequence in which statements appear. The assignment of x and y happens at particular points in that sequence, before the change the old value must be used and after the assignment the new value must be used, until it's changed again.


Example Profile Filter Links with PHP client library

Hi !
I saw that there isn't example with PHP client library on Profile Filter Links.
I had to search into sources to find methods ans her parameters.
So, I would like to help someone who's need to use Filter Links with PHP like me.
Here my code example for insert:
$filterRef = new Google_Service_Analytics_FilterRef();

$filterRef->setId('123456789');

$profileFilterLink = new Google_Service_Analytics_ProfileFilterLink();
$profileFilterLink->setFilterRef($filterRef);

$analytics->management_profileFilterLinks
    // Account ID, Property ID, View ID, Google_Service_Analytics_ProfileFilterLink
    ->insert('123456', 'UA-123456789-1', '123456789', $profileFilterLink);

And example for list:
$analytics->management_profileFilterLinks
    // Account ID, Property ID, View ID
    ->listManagementProfileFilterLinks('123456', 'UA-123456789-1', '123456789');

management_profileFilterLinks is an instance of Google_Service_Analytics_Resource_ManagementProfileFilterLinks class, defined in vendor/google/apiclient-services/src/Google/Service/Analytics/Resource/ManagementProfileFilterLinks.php.
You will find all methods into this file.

Thank you for providing this very helpful info!


Coreopts further refinement

This PR builds on knight42's bugfix of passing in package name in the constructor to also pass in version. The result of this is that the version is that of the utility itself rather than the version uucore.
Additionally, syntax, summary, and long help are now passed in as separate strings. In this way each uumain() function need not repeat their relative position. These changes offer the following the benefits:

keeps code dry, as utilities no longer have to individually create a format string for utility name, version, syntax, usage summary, and long help
better reflects the actual semantics of coreopts than the previous approach where the long help was required to use coreopts parse but this was not enforced at compile-time.

@nathanross still fails on travis

@mpkh passing

These are addressed now. I've it unsquashed so you can see the exact diff.
You can now add a fourth argument to new_coreopts! which, if false will prevent the help message from including the usage, its surrounding newlines, and the "Reference" header.

Great!


Disable running e2e tests for CertificateSigningRequests if feature gate not enabled

This PR will prevent the CertificateSigningRequest e2e tests to run if the environment variable FEATURE_GATES does no include ExperimentalCertificateSigningRequestControllers=true. This prevents tests from failing for kube versions pre 1.9.
/assign @irbekrm
NONE

/test pull-cert-manager-e2e-v1-20
/test pull-cert-manager-e2e-v1-19
/test pull-cert-manager-e2e-v1-18
/test pull-cert-manager-e2e-v1-17
/test pull-cert-manager-e2e-v1-16

Thanks Josh!
/lgtm
/approve

/kind cleanup


Is there any way I can download the Account Statement as .csv?

With the enhancement contributed in #16 you can make your own:
import csv
from datetime import date

degiro = ...  # connect & login here
ao = degiro.account_overview(date(2021, 4, 1), date.today())
cash_movements = ao["cashMovements"]  # type: List[Dict]
fieldnames = ['date', 'valueDate', 'description', 'currency', 'change', 'type', 'exchangeRate', 'orderId']
# productId, balance: also available but they would require further processing
with open('report.csv', 'w', newline='', encoding='utf-8') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames, extrasaction="ignore")
    writer.writeheader()
    writer.writerows(cash_movements)

Hummmm, I see, thanks @Jakub-CZ .
I was hoping to get something in the same shape as the .csv file available for download.
Date, Time, Value date, Product, ISIN, Description, FX, Change_Currency,  Change_Value, Balance_Currency, Balance_Value, and Order Id
Is the ISIN represented by any of the fields from fieldnames ?
What is type from fieldnames, by the way?
fieldnames = ['date', 'valueDate', 'description', 'currency', 'change', 'type', 'exchangeRate', 'orderId']

In my report types is either of these two: {'TRANSACTION', 'CASH_TRANSACTION'}. Personally, I don't use that attribute for anything.

The field productId gives you some kind of Degiro's internal number for the stock. You can translate it to something meaningful  (e.g. the ISINs) using degiro.product_info(). See https://github.com/lolokraus/DegiroAPI#product_info
This is how you can build a handy-dandy dictionary mapping all the productIds that appear in cash_movements to ISINs:
from pprint import pprint
from degiroapi.product import Product

pid_map = {p: Product(degiro.product_info(p)).isin for p in filter(None, {e.get("productId") for e in cash_movements})}
pprint(pid_map)

# {322171: 'US4581401001',
#  331916: 'US94106L1098',
#  332062: 'US7170811035',
#  ...

How to put this all together in a single CSV is left as an exercise to the reader.


Can't open more than one instance when VSCode is run as administrator

VSCode Version: 1.19.0-insider
OS Version: Windows 10 Pro

Steps to Reproduce:

Run VSCode as administrator (either by right-clicking the icon or setting it as the default in Properties -> Advanced)
Try to start another instance, either by choosing "New Window" or "Visual Studio Code - Insiders"

Reproduces without extensions: Yes
It may be worth pointing out that I'm using cmder as my integrated terminal. I have not tried disabling it to see if it fixes the issue but will try that and report back.

@joemaffei does it work if you run from the command line?

I can reproduce:
Error: connect EPERM \\.\pipe\ebe3a8355186f99da77237014959374e-1.19.0-insider-main-sock
    at Object.exports._errnoException (util.js:1050:11)
    at exports._exceptionWithHostPort (util.js:1073:20)
    at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)

You have to use different user data directories... We don't support sharing them across users.

@joaomoreno that is unfortunate. If we knew the user was running as admin, we could maybe use a different socket. Or actually we should detect this early enough that we force a different user data directory right from the beginning.
/duplicate 6560


Additional languages

Additional languages are planned to be added through the month of December 2020.
For each language, the following needs to be updated:

Verify that the new language is included in all of the translation.json files in the front-end locales folder.  For example, if adding Korean, the Japanese translations needs to include the line "Korean":"韓国語 (ko)"
The languages array in globals.js needs to be updated to include the new languages: https://github.com/OpenChain-Project/Online-Self-Certification-Web-App/blob/43f3b1859e76cf839d1e9b9242184690be645290/WebContent/resources/global.js#L24  Note that the language needs to match exactly the text in the translations used in step 2 above.

@brandelune If you could take care of 1-3 above in one or more pull requests, I'll take care of 4.
BTW - when I went and looked at the existing translations, most of the languages were already present for step 2 above - but we still should verify.  We missed Korean in our last rollout which didn't show up translated.

@brandelune I updated the tasks above to reflect the fact we are not localizing the app (sorry for the confusion).
The only thing that I need to prepare for the languages is to verify the translations already exist for the translations we currently support (item 1 above).
It looks like this may already be taken care of depending on which language your using.
Let me know if any updates or needed - or let me know the planned languages, and I'll check.

I'm only seeing item 1 and 2. I'll be getting the files later today. If you can clarify what item 3 is I'll handle it.

@brandelune - Sorry for the confusion, I updated the issue once I realized we were only localizing the questionnaire - not the entire website.

Gary, here is a csv blob for the 11 languages in the 6 primary languages (5+english). The remaining language versions will come on the 17th.
en,ja,zh-zh,zh-tw,de,ko
English,英語,英文,英語,Englisch,영어
Simplified Chinese,簡体字,简体字,中文簡體,Vereinfachtes Chinesisch,간체
Traditional Chinese,繁体字,繁体字,中文繁體,Traditionelles Chinesisch,번체
Korean,韓国語,韩文,韓語,Koreanisch,한국어
Japanese,日本語,日文,日語,Japanisch,일본어
German,ドイツ語,德文,德語,Deutsch,독일어
French,フランス語,法文,法語,Französisch,프랑스어
Italian,イタリア語,意大利文,義大利語,Italienisch,이태리어
Spanish,スペイン語,西班牙文,西班牙語,Spanisch,스페인어
Hindi,ヒンジ語,印地文,北印度語,Hindi,힌지어
Russian,ロシア語,俄文,俄語,Russisch,러시아어
Hebrew,ヘブライ語,希伯来文,希伯來語,Hebräisch,히브리어

@brandelune Thanks for the language translations - this should work.  I'll update the app tomorrow.

Added PR #201 with the additional terms.


200 Not registered error on sending group notification using GCM notification key

I am trying to send gcm group notifications to android devices from my rails server. But unfortunately I can't get it to work. But if I try to send the same message to single registration id, it works fine. So i checked it with curl-
curl -vvv -X POST --header "Content-Type: application/json" --header
"project_id: MyProjectid" --header "Authorization: key=Mykey" --data @-
"https://android.googleapis.com/gcm/send" << EOF
{
"to": "Reg_id",
"data": {}
}
EOF
The above gives me 200 OK. But here if I replace the Reg_id with the notification_key I get 200 NotRegistered and the message was not delivered. So I did a little more research and found out when I send a gcm message the notification_key get unregistered by itself.
Below is the simple scenario with I made this assumption - If I create a new notification_key using curl it gives me the notification_key. If I create a new key using the same notification_key_name I get notification_key already exists. But once I send a notification using same notification_key, I get 200 Not registered error. Then if I try to create a new notification_key using previous notification_key_name I get a new notification_key.
This seems very weird to me. Has anyone faced the same issue or found the solution for this?

Hi Andreas,
We had a recent issue in the send API, affecting a subset of the notification_key requests.
That issue should be fixed now.
Please try again. If the API is still not working for you please report back using:
https://support.google.com/code/contact/gcm_dev_support
Thanks,

Diego


grpc C# reports failure to bind port by BoundPort == 0

In the C# grpc library I'm seeing that a failure to bind/listen to a ServerPort is reported by the ServerPort.BoundPort property being 0 after calling Server.Start(). Is this the expected behavior? My expectation would have been for Server.Start() to throw an exception, based on my experience with System.Net.Sockets.TcpListener and System.Net.Sockets.Socket.
Here's a sample program that demonstrates what I'm seeing:
namespace grpc_port_bind_failure
{
    using System;
    using System.Net;
    using System.Net.Sockets;

    using Grpc.Core;

    class PortBindFailure
    {
        const int ConflictPort = 50051;

        static void Main(string[] args)
        {
            // Start a TcpListener on a port to prevent gRPC from being able to bind to it later.
            var tcpListener = new TcpListener(IPAddress.Loopback, ConflictPort);
            tcpListener.Start();
            Console.WriteLine($"tcpListener listening on {tcpListener.LocalEndpoint}");

            try
            {
                // Attempt to bind to the port--again. Expect failure, and get failure.
                var tcpListener2 = new TcpListener(IPAddress.Loopback, ConflictPort);
                tcpListener2.Start();
                Console.WriteLine($"tcpListener2 listening on {tcpListener.LocalEndpoint}!?!");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"tcpListener2 failed to start: {ex}");
            }

            var s = new Server
            {
                Ports =
                {
                    new ServerPort("<IP_ADDRESS>", ConflictPort, ServerCredentials.Insecure),
                }
            };

            // Attempt to bind to the port--again. Expect failure, but get success.
            s.Start();

            Console.WriteLine("Binding details for gRPC ports:");

            foreach (var p in s.Ports)
            {
                Console.WriteLine($"{p.Host}:{p.Port} actually listening on {p.BoundPort}");
            }
        }
    }
}

Here's the console output when I run this:
tcpListener listening on <IP_ADDRESS>:50051
tcpListener2 failed to start: System.Net.Sockets.SocketException (0x80004005): Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at System.Net.Sockets.TcpListener.Start(Int32 backlog)
   at System.Net.Sockets.TcpListener.Start()
   at grpc_port_bind_failure.PortBindFailure.Main(String[] args) in D:\TEMP\grpc-port-bind-failure\grpc-port-bind-failure\Program.cs:line 24
Binding details for gRPC ports:
<IP_ADDRESS>:50051 actually listening on 0

Environment details:

Console application
NuGet packages Grpc and Grpc.Core 1.3.0
Targeting .NET Framework 4.6.1
Visual C# 2015 (Visual C# Compiler version <IP_ADDRESS>616)
Windows 10 Version 1703 (OS Build 15063.250) amd64

By default the gRPC C# servers will start with SO_REUSEPORT. There's an option to disable this  with a "channel arg" though, an example of doing so is in https://github.com/grpc/grpc/blob/master/src/csharp/Grpc.Examples.Tests/MathClientServerTests.cs#L59

actually might have misunderstood, leaving this open to take another look at <IP_ADDRESS>:50051 actually listening on 0

Will be in v1.4.x release.


Licensing

Hi @ricksbrown ! I'm curious about the licensing of this repository. I'd like to use this project, however, I'm prohibited from leveraging this codebase if certain licenses aren't attached. Please let me know if you have plans to add licensing information to this project!

Hi @johnedquinn - thank you for taking the time to write to me. I actually have a bunch of enhancements to this codebase sitting on a local branch somewhere (if I can find it) - I think I totally threw away and re-did the unit testing because it was too complex.
Anyway, does GNU GPLv3 work for you?

Thanks for the response! Seeing that the project's last commit was some years ago, I wasn't sure if I'd get a response.
Unfortunately, GNU GPLv3 is currently restricted in the context where I'm using it -- though, I'm working on making an exception. That being said, other licenses are free to use such as: The MIT License, Apacha v2.0, Apache v1.1, and BSD licenses.

Yeah I kind of neglected this repo since I stopped using it myself.
I really should give it some love :D

I've really enjoyed testing it over the past couple days! Also, for your recent commit, you still have licensing information in your POM. BTW, is there any ETA for your release to Maven? And, if there's anything I can do, I'd be glad to submit PRs!

I've really enjoyed testing it over the past couple days! Also, for your recent commit, you still have licensing information in your POM. BTW, is there any ETA for your release to Maven? And, if there's anything I can do, I'd be glad to submit PRs!

You're brave - the doco is terrible, I can't remember how to use it.
It looks like I put a fair bit of work into it, if it's still useful then I think the first thing to "fix" is the readme :D
There seem to be some failing unit tests, I'll see if I can work that out too.
BTW it's already in maven, I added it to the readme.

Regarding Maven, the GPL license is still associated with v1.5

Regarding Maven, the GPL license is still associated with v1.5

Gotcha!
OK I'll aim to get something out in the next few days.
I need to make it worth my while, so I'll do some minor housekeeping first, like bump up a few dependency versions and move the target off java 1.6

Hi @johnedquinn FYI I released 1.6 to Maven Central today. 🎉
It's actually pretty useful, I might start using it again myself!
Note: the maven config changed a little from 1.5 - example usage is now in the readme.
Have fun with it and thanks for giving me the motivation to finally clean it up after all these years!

Thanks so much!! I'll be taking a look today!


nanoserver images are not multi-arch ones

The doc in https://hub.docker.com/_/microsoft-windows-nanoserver shows that we have multi-arch images for a range of nano images of different os versions, but after inspecting these images, I found we support only amd64 for those ones labeled with multi-arch like nano:1809
➜  ~ docker manifest inspect mcr.microsoft.com/windows/nanoserver:1809
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 574,
         "digest": "sha256:fb17f0ecdf2fc3c06d5b3b8a661f9832580f1c381de0578960cf8abc4a4e605a",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.17763.2366"
         }
      }
   ]
}

The same happens to windows-servercore https://hub.docker.com/_/microsoft-windows-servercore

I think amd64 is the only architecture supported by Windows Server for customers. So 'multiarch' here is being somewhat misused, they really mean 'manifest list', and then there's only one possible manifest to include in the list.
Maybe they're being forward-looking, and reducing friction for if-and-when they ship their ARM64 server build to the public.
It doesn't seem super-valuable to offer this for these repos, it makes more sense for things like .NET runtime where their manifest lists, e.g. 6.0, are much more extensive (although they don't use the term multiarch there, or even highlight that it's a manifest list).


Use form method instead of reading request body

Fix for https://github.com/aekasitt/fastapi-csrf-protect/issues/19

Noted with thanks. I saw some similar suggestions before and did not feel comfortable adding a version: 0.0.6 dependency to an already 0.X library. Though I understand that the lib has fastapi's support. Please check the latest commit which I feel more comfortable about for being minimally invasive, though many tests still required. (My ghetto body parsing needs work)

Works good but look at fastapi_csrf_protect/load_config.py I put correct message
'The "body_key" must be present when "token_location" is "body"'
->
'The "token_key" must be present when "token_location" is "body"'

Can you make a new PR? That's a good idea and I want to list you as contributor?

@mlojewski-me @aekasitt any reason why this wasn't merged? I see https://github.com/aekasitt/fastapi-csrf-protect/pull/21 was, but it only contains the change in the error message. Current code in master still has the same implementation for _get_csrf_from_body():
https://github.com/aekasitt/fastapi-csrf-protect/blob/1f756722c766ebae720c397826b2b25d5b944a0b/src/fastapi_csrf_protect/core.py#L58C1-L70C40
And it doesn't seem to work, at least not with latest Pydantic, as it does not (automatically) coerce a string into a dict. If the form wasn't read before the call to csrf_protect.validate_csrf(request), request._form is None and _get_csrf_from_body() fails with Input should be a valid dictionary or instance of Body.

Regarding rejecting #20  and accepting #21
We avoided CVE-2024-53981 for not adding python-multipart to our dependencies.


[DOCS] Change some tutorial text

I had a few issues following the how to install from source on linux so I addressed them here.

@tqchen

thanks @rkimball @comaniac @binarybana


Create domain script does not check that a NodePort defined in the input files is already in use

This is a usability issue noted by Antonio from CERN.  Create domain script does not check that a NodePort defined in the input files is already in use.
- It means that the script will fail just at the Kubernetes services creation time.
- It means that also the directory on the shared volume is already created. Probably it would be good to validate NodePort and catch the conflict before.

This looks quite old. Feel free to re-open if/when needed.


Cannot evaluate Aurelia string interpolation and binding expressions in cellRenderer

I'm submitting a ...  (check one with "x")
[ X] bug report => search github for a similar issue or PR before submitting
[ ] feature request
[ ] support request => Please do not submit support request here, instead see https://github.com/ceolter/ag-grid/blob/master/CONTRIBUTING.md#question

Current behavior
In a cellRenderer like the following, the Aurelia string interpolation and binding expressions are not evaluated:
            cellRenderer: function(params) {
              var eDiv = document.createElement('div');
              eDiv.innerHTML = ' ';
              var eButton = eDiv.querySelectorAll('.btn-simple')[0];

              return eDiv;
            }

Expected behavior
We need to be able to use these expressions in a cellRenderer.  Without this, we can't bind click actions to methods in Aurelia ViewModels.
Minimal reproduction of the problem with instructions
See above example.
What is the motivation / use case for changing the behavior?
See above expected behavior.
Please tell us about your environment:
N/A

ag-Grid version: X.X.X
7.2.2

Browser:
All.

Language: TypeScript

We dont support dynamic binding etc in cell renderer (or editor/filter) functions - functions are for "pure" js only.
For aurelia, you need to make use of templates (or components) - I suggest you take a look at the aurelia example project for examples of what we support here: https://github.com/ceolter/ag-grid-aurelia-example
Going forward, can you please raise issues under the https://github.com/ceolter/ag-grid-aurelia repo for aurelia related issues - thanks

@seanlandsman
Sean,
OK, I converted this piece of implementation over to use the declarative style with the Aurelia plugin, and it works fairly well.  I have run into one issue though.
Again, my goal was to use a custom renderer template in a cell.  The template looks like this:
        <ag-grid-column header-name="" >
            <ag-cell-template>

            </ag-cell-template>
        </ag-grid-column>

What's interesting, and the issue, is that the if.bind`` is being correctly evaluated and exectued; but the click.delegateis not working correctly.  The functioneditConnectionRequest()``` exists in the ViewModel (controller) but when I click the element I see this error in the console:
Uncaught Error: editConnectionRequest is not a function
    at getFunction (aurelia-binding.js:1924)
    at CallScope.evaluate (aurelia-binding.js:1527)
    at Listener.callSource (aurelia-binding.js:5064)
    at aurelia-binding.js:5088
    at HTMLDocument.handleDelegatedEvent (aurelia-binding.js:3232)

Do you have any explanation or advice for this?
This is my key use case - provide a custom cell template with clickable icons, whose click event is delegated to an Aurelia ViewModel function.
Thanks,
Don


Create Documentation for existing components using https://storybook.js.org

Storybook is an open source tool for developing UI components in isolation .It allows you to browse a component library, view the different states of each component, and interactively develop and test components. Adding Documentation for existing components using Storybook will help for future developers Currently Webui doesn't have proper documentation for existing components

Storybook documentation has been done in #184. Thanks for your contributions @ffalpha :)


[hotfix] fix valueChanges for ie11

On IE input event is not triggered on contenteditable div and valueChanges is not fired. In this fix I use MutationObserver to listen text changes.

@yuriykuzin Why do we need to uncomment pollyfills?

@yuriykuzin Why do we need to uncomment pollyfills?

Otherwise demo app doesn't start on IE11

@yuriykuzin is it better to let the consumer of the library uncomment this for IE?

@bampakoa good point, fixed. Thanks!


Enum arrays inside default arguments are not rendered using references

Describe the bug
When Enums are inside a default argument array and marked as non-nullable, they render inconsistently.
To Reproduce

Go to Codesandbox
See generated search query in documentation
See generated docs render inconsistently:

search(
  roles: [Roles!] = ["ADMIN"]
): Int!

GraphQL schema

If applicable, add an extract of the GraphQL schema.
enum Roles {
  ADMIN
  USER
}
type Query {
  search(
    roles: [Roles!] = [ADMIN]
  ): Int!
}

Expected behavior
They should render like other Enum default array arguments like this:
search(
  roles: [Roles!] = [ADMIN]
): Int!

Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

Another good catch!
Note for myself: the issue is located at https://github.com/graphql-markdown/graphql-markdown/blob/main/src/lib/graphql.js#L107

@aleccool213
The fix has been published in 1.10.2.
Working on this fix lead to discover a flaw in the mocks used by the tests that were masking a lot of issues related to lists.
I hope this new release fixes them all 🤞

Amazing, thanks again for the quick feedback :)


React-DnD 3.0.1 Critical Issue - Examples not running on Docsite

The examples are functioning locally just fine, but they are not running on the Docsite. Deprecating 3.0.1 until the doc site has been restored.

Suspect this is related to #1030

Docsite back up


The first and last parameters cause errors or are not respected

Describe the bug
The first and last parameters causes errors or are not respected.
To Reproduce
Run the program with either or both of --first  or --last.
Expected behavior
All steps should respect the first and/or last parameters and start and end at the appropriate pages.
Actual behavior
Either the entire pdf is processed or an error is raised Error: ValueError - Unknown pdf info key 'Page   22 size' in <path>
Program details (please complete the following information):

OS: Windows
Program version: v0.5.0

Fixed in v0.6.2.


Conversation Start Properties not being passed to the PVA Bot

I am passing a variable while creating the DirectLine object that I am using for my Power Virtual Agent

Like this.
However, when I am trying to access this variable in the bot topic, I am unable to.

As you see in the screenshot, I am supposed to get a "Yes" answer automatically but it is not detecting any value and the flow is moving the question topic.
I am a beginner with programming but I checked the source code on how conversationStartProperties is being used but I see that there are no references to that variable anywhere..?

Kindly help me with this issue as I was unsuccessful in finding any relevant solution anywhere on the internet.
Thank you in advance.

@kaust-technovert are you using Web Chat or are you directly using Direct Line JS in your client?
The ConversationStartProperties have a specific shape for the Start Conversation API, so it might be that what you're trying to achieve is not possible via this approach.
If you are using Web Chat, I recommend checking this sample: 04.api/a.welcome-event for sending an Event activity to the bot with relevant data before other activities begin to appear in Web Chat.

I am using DirectLineJS directly in my client, and there is also some custom implementation which I cannot mess with. So I cannot use WebChat.
Is there any example for passing ConversationStartProperties to the bot and be able to use it?

@stevengum Are you aware of any way to pass ConversationStartProperties?

@zhipeng should know more about this topic.

@kaust-technovert like Steven said "The ConversationStartProperties have a specific shape for the Start Conversation API, so it might be that what you're trying to achieve is not possible via this approach."  We do not support passing customized parameters and value for now.

Closing as not supported


CMake chokes on Doxygen (?) -   string sub-command FIND requires 3 or 4 parameters.

I cannot diagnose what the issue is here, but my guess is that the version of Doxygen I have installed on this machine is too old.
How do I disable documentation building?  I have no need for that on this system.
[jrhammon@esgmonster build]$ rm -rf *
[jrhammon@esgmonster build]$     cmake ..              -DCMAKE_CC_COMPILER="$PRK_CC"              -DCMAKE_CXX_COMPILER="$PRK_CXX"              -DMPI_C_COMPILER="\"MPICH_CC=$PRK_CC $MPI_ROOT/bin/mpicc\""              -DMPI_C_LINK_FLAGS="-L$MPI_ROOT/lib"              -DMPI_C_LIBRARIES="-lmpi"              -DMPI_C_INCLUDE_PATH="$MPI_ROOT/include"              -DMPI_CXX_COMPILER="\"MPICH_CXX=$PRK_CXX $MPI_ROOT/bin/mpicxx\""              -DMPI_CXX_LINK_FLAGS="-L$MPI_ROOT/lib"              -DMPI_CXX_LIBRARIES="-lmpicxx -lmpi"              -DMPI_CXX_INCLUDE_PATH="$MPI_ROOT/include"              -DGRAPPA_INSTALL_PREFIX=/tmp/grappa
-- The C compiler identification is GNU 4.4.7
-- The CXX compiler identification is Intel <IP_ADDRESS>50815
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /opt/intel/compilers_and_libraries_2016.0.109/linux/bin/intel64/icpc
-- Check for working CXX compiler: /opt/intel/compilers_and_libraries_2016.0.109/linux/bin/intel64/icpc -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Warning at CMakeLists.txt:44 (message):
  You are using an unsupported compiler! Compilation has only been tested
  with Clang and GCC.

-- Found MPI_C: -lmpi
-- Found MPI_CXX: -lmpicxx -lmpi
-- Boost found:  1.55.0 --
-- Downloading third-party dependences from the web.
-- Using {,} to build third-party dependencies (if needed).
     boost: /usr/local/include and /usr/local/lib
    gflags: <building>
      glog: <building>
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Found Doxygen: /usr/bin/doxygen (found version "1.6.1")
CMake Error at CMakeLists.txt:524 (string):
  string sub-command FIND requires 3 or 4 parameters.

CMake Error at CMakeLists.txt:525 (string):
  string sub-command FIND requires 3 or 4 parameters.

-- Configuring incomplete, errors occurred!
See also "/home/jrhammon/Work/PGAS/GRAPPA/github/build/CMakeFiles/CMakeOutput.log".

My version of CMake is sufficient for Grappa, according to the documentation:
[jrhammon@esgmonster build]$ cmake --version
cmake version <IP_ADDRESS>

Looks like you have empty variables in a couple places we didn't expect. Pull request #258 should take care of these errors, but we may find more.

Okay, master is updated with a fix for this. See if it works for you.


Make SPMTestSupport not require the command line tools (so it can be used for libSwiftPMDataModel)

Remove a dependency of the SPMTestSupport target on the Command target, which in turn depends on Build, etc.  This lets the subset of unit tests that doesn't actually require the command line tools to still be used to test libSwiftPMDataModel.
The only real code change needed is to move two tests that are actually testing handling of command line arguments of the swift-build tool from BuildPlanTests to BuildToolTests.  This is where it belonged anyway, architecturally, since it's testing command line arguments.

@swift-ci please smoke test


Fix go vet warning

Warning was:
github.com/golang/protobuf/proto/all_test.go:1284: result of fmt.Sprintf call not used

No thanks. The code isn't wrong.


Update module.json

Don't know what's up with the merge conflicts, but I'll just integrate this locally


Fix bug: date being changed to 2001 when marking a task as complete

Finished this yesterday right before I logged off.


Fake timers created inside coroutines which are then closed by coroutine.close/task.cancel still attempt to resume

This code causes the test to error with 'cannot spawn non-suspended coroutine with arguments', despite being valid when using real timers:
it("timer resume bug", function()
	jest.useFakeTimers()

	local thread
	task.spawn(function()
		task.wait()
		coroutine.close(thread)
	end)
	thread = task.spawn(function()
		task.wait()
		-- no code should execute as this point, and there shouldn't be any errors
	end)

	jest.runAllTimers()
end)

If coroutine.close is swapped with task.cancel, any code after the task.wait() in the thread will still execute, whereas with real timers it won't.

This patch fixes it for me:
---<EMAIL_ADDRESS>+++<EMAIL_ADDRESS>@@ -235,6 +235,7 @@
 	local delayTimeMs = (self._engineFrameTime / 1000) + delayTime * 1000
 	local targetTime = self._mockTimeMs + delayTimeMs
 	local timeout = {
+		thread = coroutine.running(),
 		time = targetTime,
 		callback = callback,
 		args = { ... },
@@ -254,9 +255,18 @@
 	return timeout
 end

-local function fakeCancel(self, timeout)
-	for i, timeout_ in self._timeouts do
-		if timeout_ == timeout then
+local function fakeCancel(self, timeoutOrThread)
+	if typeof(timeoutOrThread) == "thread" then
+		realTask.cancel(timeoutOrThread)
+	end
+	for i, timeout in self._timeouts do
+		if timeout == timeoutOrThread or timeout.thread == timeoutOrThread then
+			-- Some timer advancement functions clone the _timeouts array and
+			-- work on that, which we can't modify here. So we also need to
+			-- clear the callback/args to make sure it doesn't do anything
+			timeout.callback = function() end
+			timeout.args = {}
+			timeout.time = self._mockTimeMs
 			table.remove(self._timeouts, i)
 			break
 		end
@@ -267,6 +277,12 @@
 	local running = coroutine.running()
 	local clock = fakeClock(self)
 	fakeDelay(self, timeToWait or 0, function()
+		if coroutine.status(running) == "dead" then
+			-- This was probably canceled by coroutine.close, which we don't
+			-- mock. We can just treat it like task.cancel and return
+			fakeCancel(self, running)
+			return
+		end
 		task.spawn(running, fakeClock(self) - clock)
 	end)
 	return coroutine.yield()


Add non-implemented get_deleted_bonds interface.

Adds interface requested in issue #219
Currently, no reactions actually use this.

I will also add a reaction that does delete bonds to test this.

I propose this is not so straight forward and requires many more changes:

an interface for the RingAmineReaction is required - currently it cannot be used through a Factory.
A BreakBondReactionFactory is required - this will be used in-place of a GenericReactionFactory
FunctionalGroup API needs updating to be able to define an FG that can lose bonds (currently deleters and bonders cannot define this without breaking their meaning) - this is not actually true, just need a new FunctionalGroup class, akin to GenericFunctionalGroup, which handles this API ->
a Functional Group (eg. BreakBond and BreakBondFactory), which take in a Smarts string to define a sub structure and bond-to-be-broken atom ids (whatever we want to call them)

Therefore, I suggest that adding a use of this and test will be a later PR alongside these changes.


How to update or delete zuul routes dynamically?

Hi
I am using Jhipster gateway. I want to facilitate UI features to my clients to add, update and delete the routes(which is basically external API from the clients). I found an option to update application.yml file to add zuul routes. However, there is no option to delete it dynamically via class such as DiscoveryClientRouteLocator which have the option of adding the routes but not deletion or updation. Is this feature is implemented? Any help would be appreciated. Thanks

Guys, I've got it. I used zuulproperties.getRoutes().remove(zuulRoute). It worked. Thanks


Add here map widget for jupyter as one of the backend.

Description
I think here map widget for jupyter. will be good addition to this library.
Let me know if you want to add here map widget for jupyter in leafmap, I am happy to contribute.

That would be awesome! Pull requests are welcome.

closing this as implementation is done in #20


Inspect and optimize memory use

I have the idea that the explorer is using a lot of memory and I suspect we have some memory leaks too. (I've been doing some profiling using Google Chrome's timeline in the developer toolbox).
However, this area is new to me.

We are sunsetting the bird tracking explorer (+ Carto accounts only accepts limited API calls). I'm closing any related issues.


Live preview doesn't work on Firefox/Mac

- Bug -
The live preview flashes up and then disappears on Firefox. No errors in browser console.

Here is the console message I get in latest Firefox on MacOS:
 Checking for MetaData files  cms.js:41:30228
 2017-03-17-smashing-mag-va-dix-fois-plus-vite does not have metadata  cms.js:41:30551

So it seems this is an iframe issue - removing the iframe from the equation works. A developer from Zendesk ran this whole React iframe rendering gauntlet back in 2014 and created react-frame-component to as a packaged solution. I think our next step is to give that a shot.


BLE Partial Flashing for MicroPython

This is good timing! I started work on the app today.
Could the python builder generate an 8 byte hash? The PF protocol transmits the hash from micro:bit to client in a struct with 8 bytes, to be checked against the hex. It looks like it's using hash type 2, which is a pointer to what looks like a long string, in which the first 8 bytes are always the same.

This is good timing! I started work on the app today.
Could the python builder generate an 8 byte hash? The PF protocol transmits the hash from micro:bit to client in a struct with 8 bytes, to be checked against the hex. It looks like it's using hash type 2, which is a pointer to what looks like a long string, in which the first 8 bytes are always the same.

Could the python builder generate an 8 byte hash? The PF protocol transmits the hash from micro:bit to client in a struct with 8 bytes, to be checked against the hex. It looks like it's using hash type 2, which is a pointer to what looks like a long string, in which the first 8 bytes are always the same.

To update the hash for the filesystem region? Yes, I probably should update it although I'm not checking it anywhere at the moment

Could the python builder generate an 8 byte hash? The PF protocol transmits the hash from micro:bit to client in a struct with 8 bytes, to be checked against the hex. It looks like it's using hash type 2, which is a pointer to what looks like a long string, in which the first 8 bytes are always the same.

To update the hash for the filesystem region? Yes, I probably should update it although I'm not checking it anywhere at the moment

addlayouttable.py doesn't provide a hash for the filesystem region, and I don't check it either.
I'm referring to the mp/DAL region hash.
My understanding is that addlayouttable.py uses a pointer hash for the mp region, which is the equivalent of the DAL region for MakeCode, and is the hash we check, before a partial flash: https://github.com/microbit-foundation/micropython-microbit-v2/blob/master/src/addlayouttable.py#L211.
The mp/DAL region hash pointer points to microbit_version_string, which is initialised with MICROBIT_VERSION
https://github.com/microbit-foundation/micropython-microbit-v2/blob/a76e1413bcd66f128a31d98756fc3d1f336d1580/src/codal_port/modos.c#L36,
and is something like "micro:bit v2.0.0-beta.2+47ebc73-dirty on 2021-01-05; MicroPython v1.13 on 2021-01-05".
https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/MicroBitMemoryMap.cpp#L176
processRecord copies 8 bytes from the string pointed to by the type 2 hash pointer. The first 8 bytes of that string "micro:bi" are no use. A unique version number or date & time might be OK, I suppose. Sending more that 8 bytes of hash will need a change to the protocol.
I have partial flashing working in the app with the existing PF service, by temporarily ignoring the hash, and instead checking the start and end addresses of the mp/dal region when the incoming hex is micropython. I'm using the filesystem region in the hex's table to decide what to send.
I measure (counting in my head!) a full flash at 66s and a partial flash as 24s, which feels very slow for a minimal hello world. Could the filesystem region (currently 6D000 to 73000) be made more compact and accurate, to just include the current script files?

addlayouttable.py doesn't provide a hash for the filesystem region, and I don't check it either.
I'm referring to the mp/DAL region hash.
My understanding is that addlayouttable.py uses a pointer hash for the mp region, which is the equivalent of the DAL region for MakeCode, and is the hash we check, before a partial flash: https://github.com/microbit-foundation/micropython-microbit-v2/blob/master/src/addlayouttable.py#L211.
The mp/DAL region hash pointer points to microbit_version_string, which is initialised with MICROBIT_VERSION
https://github.com/microbit-foundation/micropython-microbit-v2/blob/a76e1413bcd66f128a31d98756fc3d1f336d1580/src/codal_port/modos.c#L36,
and is something like "micro:bit v2.0.0-beta.2+47ebc73-dirty on 2021-01-05; MicroPython v1.13 on 2021-01-05".
https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/MicroBitMemoryMap.cpp#L176
processRecord copies 8 bytes from the string pointed to by the type 2 hash pointer. The first 8 bytes of that string "micro:bi" are no use. A unique version number or date & time might be OK, I suppose. Sending more that 8 bytes of hash will need a change to the protocol.
I have partial flashing working in the app with the existing PF service, by temporarily ignoring the hash, and instead checking the start and end addresses of the mp/dal region when the incoming hex is micropython. I'm using the filesystem region in the hex's table to decide what to send.
I measure (counting in my head!) a full flash at 66s and a partial flash as 24s, which feels very slow for a minimal hello world. Could the filesystem region (currently 6D000 to 73000) be made more compact and accurate, to just include the current script files?

Getting back to this

processRecord copies 8 bytes from the string pointed to by the type 2 hash pointer. The first 8 bytes of that string "micro:bi" are no use. A unique version number or date & time might be OK, I suppose. Sending more that 8 bytes of hash will need a change to the protocol.

I hadn't spotted this.. thanks Martin

I measure (counting in my head!) a full flash at 66s and a partial flash as 24s, which feels very slow for a minimal hello world. Could the filesystem region (currently 6D000 to 73000) be made more compact and accurate, to just include the current script files?

For partial flashing are you flashing the whole file system? I'm skipping blank pages after the script, as CODAL will erase the region from the last page written in flashing to the end of the region

Getting back to this

processRecord copies 8 bytes from the string pointed to by the type 2 hash pointer. The first 8 bytes of that string "micro:bi" are no use. A unique version number or date & time might be OK, I suppose. Sending more that 8 bytes of hash will need a change to the protocol.

I hadn't spotted this.. thanks Martin

I measure (counting in my head!) a full flash at 66s and a partial flash as 24s, which feels very slow for a minimal hello world. Could the filesystem region (currently 6D000 to 73000) be made more compact and accurate, to just include the current script files?

For partial flashing are you flashing the whole file system? I'm skipping blank pages after the script, as CODAL will erase the region from the last page written in flashing to the end of the region

For partial flashing are you flashing the whole file system? I'm skipping blank pages after the script, as CODAL will erase the region from the last page written in flashing to the end of the region

Yes :( After finding the partial flash region, I was forgetting check what part of that was actually filled by the hex!

For partial flashing are you flashing the whole file system? I'm skipping blank pages after the script, as CODAL will erase the region from the last page written in flashing to the end of the region

Yes :( After finding the partial flash region, I was forgetting check what part of that was actually filled by the hex!

https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/bluetooth/MicroBitPartialFlashingService.cpp#L325
It doesn't look like the calculated flashPointer will always be on a page boundary.
https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/bluetooth/MicroBitPartialFlashingService.cpp#L344
This fixes up the settings pages after the PF in case it had a CRC validation, which the bootloader creates after a DFU. I overlooked the fact that we need to save no-validation settings pages before starting PF. Otherwise if PF fails the bootloader will drop into DFU mode. And if repeated partial flashings are like likely, it might be better to leave it as no-validation.

https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/bluetooth/MicroBitPartialFlashingService.cpp#L325
It doesn't look like the calculated flashPointer will always be on a page boundary.
https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/bluetooth/MicroBitPartialFlashingService.cpp#L344
This fixes up the settings pages after the PF in case it had a CRC validation, which the bootloader creates after a DFU. I overlooked the fact that we need to save no-validation settings pages before starting PF. Otherwise if PF fails the bootloader will drop into DFU mode. And if repeated partial flashings are like likely, it might be better to leave it as no-validation.

It doesn't look like the calculated flashPointer will always be on a page boundary.

I've added a mask to find the page boundary

This fixes up the settings pages after the PF in case it had a CRC validation, which the bootloader creates after a DFU. I overlooked the fact that we need to save no-validation settings pages before starting PF. Otherwise if PF fails the bootloader will drop into DFU mode. And if repeated partial flashings are like likely, it might be better to leave it as no-validation.

Ok good idea, so should the settings code be moved to where the KeyValue flag is set? Then a failed partial flash will enter pairing mode
I wonder if this should behave differently on Python, if the partial flash fails and MicroPython starts with a corrupt filesystem would the REPL still work..

It doesn't look like the calculated flashPointer will always be on a page boundary.

I've added a mask to find the page boundary

This fixes up the settings pages after the PF in case it had a CRC validation, which the bootloader creates after a DFU. I overlooked the fact that we need to save no-validation settings pages before starting PF. Otherwise if PF fails the bootloader will drop into DFU mode. And if repeated partial flashings are like likely, it might be better to leave it as no-validation.

Ok good idea, so should the settings code be moved to where the KeyValue flag is set? Then a failed partial flash will enter pairing mode
I wonder if this should behave differently on Python, if the partial flash fails and MicroPython starts with a corrupt filesystem would the REPL still work..

should the settings code be moved to where the KeyValue flag is set?

I reckon that's right. Get rid of the validation fix up at the end and ensure noValidation just before storing the flashIncomplete flag:
https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/bluetooth/MicroBitPartialFlashingService.cpp#L292
That way only a failed full DFU flash will end up stuck in DFU mode.
void noValidation()
{
    // make bootloader settings page
    nrf_dfu_settings_t settings;
    memset( &settings, 0, sizeof( nrf_dfu_settings_t));

    // make NO_VALIDATION settings
    settings.settings_version = 2;
    settings.bank_0.bank_code = NRF_DFU_BANK_VALID_APP;
    settings.boot_validation_app.type = NO_VALIDATION;

    // make VALIDATE_CRC settings - calculate app size and CRC
    //settings.boot_validation_app.type = VALIDATE_CRC;
    //settings.bank_0.image_size = offset + 16 * sizeof( uint32_t) - MICROBIT_APP_REGION_START;
    //settings.bank_0.image_crc  = crc32_compute( (uint8_t*) MICROBIT_APP_REGION_START, settings.bank_0.image_size, NULL);
    //memcpy(settings.boot_validation_app.bytes, &settings.bank_0.image_crc, sizeof(uint32_t));

    // calculate settings page CRCs
    settings.crc = crc32_compute( (const uint8_t *)&settings.settings_version,
                                  offsetof(nrf_dfu_settings_t, init_command) - offsetof(nrf_dfu_settings_t, settings_version),
                                  NULL);

    settings.boot_validation_crc = crc32_compute( (const uint8_t *)&settings.boot_validation_softdevice, 3 * sizeof(boot_validation_t), NULL);

    uint32_t settingsSizeInWords = ( sizeof( nrf_dfu_settings_t) + sizeof(uint32_t) - 1) / sizeof(uint32_t);

    // save settings to flash settings page if different
    uint32_t *pSettings = (uint32_t *) MICROBIT_BOOTLOADER_SETTINGS;
    if ( memcmp( pSettings, &settings, sizeof( nrf_dfu_settings_t)))
    {
      flash.erase_page( pSettings);
      flash.flash_burn( pSettings, (uint32_t *)&settings, settingsSizeInWords);
    }

    // save settings to flash settings backup page if different
    uint32_t *pBackup   = (uint32_t *) MICROBIT_MBR_PARAMS;
    if ( memcmp( pBackup, &settings, sizeof( nrf_dfu_settings_t)))
    {
      flash.erase_page( pBackup);
      flash.flash_burn( pBackup, (uint32_t *)&settings, settingsSizeInWords);
    }
}

should the settings code be moved to where the KeyValue flag is set?

I reckon that's right. Get rid of the validation fix up at the end and ensure noValidation just before storing the flashIncomplete flag:
https://github.com/lancaster-university/codal-microbit-v2/blob/14f2a243ca869a56c10d5bb80e24c68d80290ab0/source/bluetooth/MicroBitPartialFlashingService.cpp#L292
That way only a failed full DFU flash will end up stuck in DFU mode.
void noValidation()
{
    // make bootloader settings page
    nrf_dfu_settings_t settings;
    memset( &settings, 0, sizeof( nrf_dfu_settings_t));

    // make NO_VALIDATION settings
    settings.settings_version = 2;
    settings.bank_0.bank_code = NRF_DFU_BANK_VALID_APP;
    settings.boot_validation_app.type = NO_VALIDATION;

    // make VALIDATE_CRC settings - calculate app size and CRC
    //settings.boot_validation_app.type = VALIDATE_CRC;
    //settings.bank_0.image_size = offset + 16 * sizeof( uint32_t) - MICROBIT_APP_REGION_START;
    //settings.bank_0.image_crc  = crc32_compute( (uint8_t*) MICROBIT_APP_REGION_START, settings.bank_0.image_size, NULL);
    //memcpy(settings.boot_validation_app.bytes, &settings.bank_0.image_crc, sizeof(uint32_t));

    // calculate settings page CRCs
    settings.crc = crc32_compute( (const uint8_t *)&settings.settings_version,
                                  offsetof(nrf_dfu_settings_t, init_command) - offsetof(nrf_dfu_settings_t, settings_version),
                                  NULL);

    settings.boot_validation_crc = crc32_compute( (const uint8_t *)&settings.boot_validation_softdevice, 3 * sizeof(boot_validation_t), NULL);

    uint32_t settingsSizeInWords = ( sizeof( nrf_dfu_settings_t) + sizeof(uint32_t) - 1) / sizeof(uint32_t);

    // save settings to flash settings page if different
    uint32_t *pSettings = (uint32_t *) MICROBIT_BOOTLOADER_SETTINGS;
    if ( memcmp( pSettings, &settings, sizeof( nrf_dfu_settings_t)))
    {
      flash.erase_page( pSettings);
      flash.flash_burn( pSettings, (uint32_t *)&settings, settingsSizeInWords);
    }

    // save settings to flash settings backup page if different
    uint32_t *pBackup   = (uint32_t *) MICROBIT_MBR_PARAMS;
    if ( memcmp( pBackup, &settings, sizeof( nrf_dfu_settings_t)))
    {
      flash.erase_page( pBackup);
      flash.flash_burn( pBackup, (uint32_t *)&settings, settingsSizeInWords);
    }
}

I wonder if this should behave differently on Python, if the partial flash fails and MicroPython starts with a corrupt filesystem would the REPL still work.

I expected MicroPython would check the flashIncomplete flag, and boot to pairing mode to be consistent. If MicroPython can recognise a corrupt filesystem and not crash, I guess that would be OK, but only potentially useful if you can use the REPL. The no-validation settings would still be required, but maybe the flashIncomplete shouldn't be saved. What happens if the filesystem is changed via USB after a BLE full flash?

I wonder if this should behave differently on Python, if the partial flash fails and MicroPython starts with a corrupt filesystem would the REPL still work.

I expected MicroPython would check the flashIncomplete flag, and boot to pairing mode to be consistent. If MicroPython can recognise a corrupt filesystem and not crash, I guess that would be OK, but only potentially useful if you can use the REPL. The no-validation settings would still be required, but maybe the flashIncomplete shouldn't be saved. What happens if the filesystem is changed via USB after a BLE full flash?

Sorry @microbit-sam. I think I might be wrong about the bootloader validation needing to be disabled before a partial flash. It's a while since I looked at it, but I have a feeling the validation only runs during the first reboot after a full DFU. The reason we invented a no validation setting was so that the application size didn't need to be customised when building a hex. No validation is interpreted as "size is not valid - do single bank dfu". So back to the way it was, I think, except putting it in a function is good.

Sorry @microbit-sam. I think I might be wrong about the bootloader validation needing to be disabled before a partial flash. It's a while since I looked at it, but I have a feeling the validation only runs during the first reboot after a full DFU. The reason we invented a no validation setting was so that the application size didn't need to be customised when building a hex. No validation is interpreted as "size is not valid - do single bank dfu". So back to the way it was, I think, except putting it in a function is good.

@microbit-sam @microbit-carlos In a hello world python hex, the filesystem region has 256 bytes at the beginning (0x6d000), including some 0xFF bytes, and a lone byte at 0x72000. I guess a partial flash has to write pages 0x6d000 and 0x72000. Does it have to ensure the pages between, are erased?

and a lone byte at 0x72000.

Yes, this is the filesystem scratch page.

Does it have to ensure the pages between, are erased?

Yes, otherwise if there were old files there they could be found by the filesystem.

So either the client app must send a block of 0xFFs for each empty page or the PF service needs to notice that pages are skipped and ensure they are blank.

Yes, I think we agreed CODAL should notice it's the fs area and erase everything that is not 0xFF already. Is that the case @microbit-sam?

Yeah, that's what I'm aiming to do in this PR on line 382:
https://github.com/lancaster-university/codal-microbit-v2/pull/64/files#diff-aa2c3a0dceda8363e676a9836986800076f7bac72b3b9e99f354c8f09aba45deR382
Hopefully that makes sense and I've not messed it up!

I don't think the erasing code in the END_OF_TRANSMISSION case will do anything because of the byte in page 0x72000. The client will start at 0x6D000 and finish at 0x72000. The END_OF_TRANSMISSION code will start looking at 0x73000 and stop immediately.
As I understand it, the pages in between will not be touched because the FLASH_DATA code erases a page only if it receives a block of packets with offset on the page boundary.
At the moment, if the client has any data in a page, it must send something on the page boundary to trigger the erase, and it must send FFs for each empty page, if the middle pages have to be blank.

Hmm.. yeah that is annoying
Perhaps CODAL needs to be aware that when partial flashing micropython it must keep track of where it is before writing the layout table. A bigger change than I'd hoped for
Looking at the records that micropython currently uses, the layout table does not change between partial flashes
Regions 1 and 2 will remain in the same space with the same hashes, and the filesystem will stay in the same space and does not contain a hash
We could ignore the layout table in partial flashing and finish when data in the file system runs out, and erase from there until the layout table. This sounds like a bad idea but it would be simpler..
Python Layout Record. Record: 3 Hash Type: 0 Start: 0x6D000 Length: 0x6000 Hash: 0x0 0x0
Python Layout Record. Record: 2 Hash Type: 2 Start: 0x1C000 Length: 0x47664 Hash: 0x0 0x5B368
Python Layout Record. Record: 1 Hash Type: 0 Start: 0x1000 Length: 0x1B000 Hash: 0x0 0x0

before writing the layout table.
Do you mean the "filesystem scratch page" 0x72000?
Yes, I think we want to avoid specialising the code for micro-python, but generalise is so it can cope with both.

We haven't dealt with sparse data before. It would be good to specify exactly what the app must send. At the moment, I 've modified the iOS app code to step through the region in 64 byte blocks (so not crossing a page boundary), sending a block (in 4 packets) if there are any non-FF bytes in it, or it's on a page boundary.
This seems like the simplest solution, except maybe FLASH_DATA should not erase the page if it's already blank.
https://github.com/lancaster-university/codal-microbit-v2/blob/50b8d901535147800ff5727ea5591e6f044d5b49/source/bluetooth/MicroBitPartialFlashingService.cpp#L347
But I think that means the app is sending around double the number of packets it needs to for "hello world", in order to erase the blank pages.
Could the PF service remember the next page that will need to be erased (initially the first page in the region), then before each write in FLASH_DATA, erase all the non-blank pages from that "next" page up to the data's page?
And finish up by erasing from that "next" page up to the last page in the region? Could the micropython filesystem scratch page be empty sometimes? Does erasing to the end of the region make sense for MakeCode?

I've added a commit to erase pages between last page written + 1, and the current page
However I think this would fall over if the packets were out of order

This seems like the simplest solution, except maybe FLASH_DATA should not erase the page if it's already blank.

Yep I've got a CRC in there to check for blank pages

The latest commit creates a table of pages, and marks which pages have been touched
pagesTouched = (uint8_t *)malloc(MICROBIT_CODESIZE);
And then erases untouched pages when partial flashing completes
      // Erase untouched pages in filesystem
      int start = (FLASH_PROGRAM_END / MICROBIT_CODEPAGESIZE);
      int end   = (MICROBIT_APP_REGION_END / MICROBIT_CODEPAGESIZE);
      for(int page = start; page < end; page++) {
        DMESG("Checking page: %u", page);
        if(!pagesTouched[page]) {
            DMESG("Erasing page: %u", page);
            flash.erase_page( (uint32_t *) (page * MICROBIT_CODEPAGESIZE));
        }
      }

Maybe pagesTouched should only be (MICROBIT_APP_REGION_END - FLASH_PROGRAM_END) long

I have a horrible feeling we have invented a really complicated way to erase all the pages from the start of the PF region to the end of the app region! If all pages that do not contain data must be blank, and all pages that do contain data must be blank in order to burn the data, isn't that all the pages?
The dal/codal code in master only changes a "no-data" page (without an erase) if it finds the MakeCode embedded script magic. Why was that necessary? Why is erasing all pages now necessary?  Will this change the use of MicroBitFileSystem?
The check for "already blank" has gone:
https://github.com/lancaster-university/codal-microbit-v2/blob/944e74693f29c0424caec191d7d1f9da9190d1c7/source/bluetooth/MicroBitPartialFlashingService.cpp#L353
https://github.com/lancaster-university/codal-microbit-v2/blob/944e74693f29c0424caec191d7d1f9da9190d1c7/source/bluetooth/MicroBitPartialFlashingService.cpp#L394
Depending on the value of blockNum, I think the following might burn the same data again or some junk or to a page that hasn't been erased yet, or to an invalid offset.
https://github.com/lancaster-university/codal-microbit-v2/blob/944e74693f29c0424caec191d7d1f9da9190d1c7/source/bluetooth/MicroBitPartialFlashingService.cpp#L378

Maybe pagesTouched should only be (MICROBIT_APP_REGION_END - FLASH_PROGRAM_END) long

Yes, or the number of pages in the PF region. A micropython PF only potentially touches 6 pages out of 128, but it would be bigger for a MakeCode app.

  int start = (FLASH_PROGRAM_END / MICROBIT_CODEPAGESIZE);
  int end   = (MICROBIT_APP_REGION_END / MICROBIT_CODEPAGESIZE);

https://github.com/lancaster-university/codal-microbit-v2/blob/944e74693f29c0424caec191d7d1f9da9190d1c7/source/bluetooth/MicroBitPartialFlashingService.cpp#L388
Shouldn't this be working on the PF region? Could this erase the last page of the application code and the micropython region table?
The pagesTouched array is only needed during partial flashing in pairing mode. Is it safe that it's only initialised in the constructor? The amount of heap available in pairing mode doesn't depend on the app, so wouldn't it be OK to lazily allocate and initialise pagesTouched, block and the other PF-only variables when needed? Then it wouldn't matter that pagesTouched was unnecessarily large.

isn't that all the pages?

facepalm Yes, of course

The dal/codal code in master only changes a "no-data" page (without an erase) if it finds the MakeCode embedded script magic. Why was that necessary? Why is erasing all pages now necessary? Will this change the use of MicroBitFileSystem?

Previously when the MakeCode source was embedded in the hex file it would invalidate the source so that if the hex was extracted, it would no longer load into MakeCode. We don't need this in CODAL
So best method would be to erase the filesystem at the start of partial flashing, then write data wherever it needs to go, and not have to worry about erasing pages during/after
Looking at MicroBitFileSystem it currently has a format function, that doesn't erase pages. Adding a parameter to also erase makes sense to me?
https://github.com/lancaster-university/codal-microbit-v2/blob/de856f22f7fbea1145c24fff7933dd12fc30b2ae/source/MicroBitFileSystem.cpp#L250

I'm not sure what you mean about MicroBitFileSystem. I was just wondering if it was generally preserved through a MakeCode partial flash before, but erasing all pages would rule that out and increase wear on the flash. Currently, a MakeCode partial flash doesn't erase any pages after the PF.

I'm not sure what you mean about MicroBitFileSystem. I was just wondering if it was generally preserved through a MakeCode partial flash before, but erasing all pages would rule that out and increase wear on the flash. Currently, a MakeCode partial flash doesn't erase any pages after the PF.

I'm thinking in terms of Python. The Python script(s) that execute are stored in the filesystem, including the main.py that the program enters to start. So for a Python partial flash I think we have to erase the whole filesystem to ensure there's no scripts left
Erasing the filesystem using MicroBitFileSystem rather than manually erasing pages would also mean that if the filesystem (or part of it) was moved to the KL27 in the future it would be handled correctly already
For MakeCode, I'm not sure whether erasing the filesystem is the right thing to do. It would be cool if you could still access any data you may have logged, but also it might be completely irrelevant!

I've changed it to reformat the filesystem when partially flashing

@microbit-sam does that mean you think this is ready for being merged then?

I think so, but as the previous comment thread demonstrates there may be something I've not considered
One worry I had was that erasing all of the pages in the filesystem might take so long that it causes an issue with the first packet. This doesn't seem to be the case when I've been using it (I ran it forcing all pages to be erased with no problems), but if the filesystem was larger using the KL27's free space maybe it'd be slower and cause problems

I still don't understand the need to erase all the trailing pages for a MakeCode PF. They didn't need to be erased before.
I didn't understand the link with MicroBitFileSystem or moving it to the KL27. Will MicroPython's file system move there too? In any case MicroPython doesn't use MicroBitFileSystem, does it?
MicroBitFileSystem:reformat() formats the filesystem, which I think means that a page or two will no longer be blank, and need to be erased again to PF. Isn't it in danger of deleting the MicroPython memory map table? I don't think MicroBitFileSystem has to have all pages erased to be reinitialised.
Is all-FFs the only combination of bytes that has that CRC?

I still don't understand the need to erase all the trailing pages for a MakeCode PF. They didn't need to be erased before.

Previously we only had to remove the embedded source, and now we don't. I think I've removed all the trailing MakeCode erasing code now (apart from the filesystem reformat)

I didn't understand the link with MicroBitFileSystem or moving it to the KL27. Will MicroPython's file system move there too? In any case MicroPython doesn't use MicroBitFileSystem, does it?

MicroPython stores it's scripts and executes from the file system
https://github.com/microbit-foundation/micropython-microbit-v2/blob/a76e1413bcd66f128a31d98756fc3d1f336d1580/src/codal_port/main.c#L49

MicroBitFileSystem:reformat() formats the filesystem, which I think means that a page or two will no longer be blank, and need to be erased again to PF. Isn't it in danger of deleting the MicroPython memory map table? I don't think MicroBitFileSystem has to have all pages erased to be reinitialised.

Erasing the filesystem should preserve the layout table
SoftDevice    0x01000..0x1c000
MicroPython   0x1c000..0x63a58
Layout table  0x63fc0..0x64000
Filesystem    0x6d000..0x73000

MBFS::init has built in checks to prevent it from overwriting an existing filesystem
and int MicroBitFileSystem::format() states
* Initialises a new file system. Assumes all pages are already erased.
I thought it made sense to create a seperate function to erase and then call format rather the modify the existing parts. But maybe for partial flashing reformating isn't necessary (as you said, the first page will be erased almost instantly), and just having an erase all function to clear the filesystem is better

Is all-FFs the only combination of bytes that has that CRC?

It's a good question, and I'm not certain how to verify. Searching the internet for the hex string gave me two results, one a forum thread about blank pages, and one some source from VMWare also searching for blank pages
However this has made me realise that 0xc71c0011 is for an all 0s page, not an all 0xFF page and will need to be changed

Previously we only had to remove the embedded source, and now we don't.

So why erase all those pages that didn't need to be touched before?

MicroPython stores it's scripts and executes from the file system

I think micropython has it's own file system and does not use MicroBitFileSystem.
https://github.com/microbit-foundation/micropython-microbit-v2/blob/a76e1413bcd66f128a31d98756fc3d1f336d1580/src/codal_port/main.c#L56
https://github.com/microbit-foundation/micropython-microbit-v2/blob/a76e1413bcd66f128a31d98756fc3d1f336d1580/src/codal_port/main.c#L39
https://github.com/microbit-foundation/micropython-microbit-v2/blob/master/.gitmodules#L3
https://github.com/micropython/micropython/blob/master/ports/nrf/modules/uos/microbitfs.c#L180

Erasing the filesystem should preserve the layout table

The micropython table is at the end of the page containing FLASH_PROGRAM_END/mp_end or the next page if there's no room in that page.
https://github.com/microbit-foundation/micropython-microbit-v2/blob/master/src/addlayouttable.py#L202
https://github.com/microbit-foundation/micropython-microbit-v2/blob/master/src/addlayouttable.py#L218
In the example, the layout table happens to be in the same page as FLASH_PROGRAM_END. MicroBitFileSystem::reformat starts erasing at  the first page boundary at or after FLASH_PROGRAM_END, which could contain the layout table in a future micropython build.
https://github.com/lancaster-university/codal-microbit-v2/pull/64/files#diff-9d1778509c3b5f7217f70a176bd7c06c1b756893999ad3d56ab38f8bf71dd303R251
In any case, reformat() calls format(), which saves data in at least one page, so it's no longer blank.
I think it would also be wise to avoid trampling on the space between the layout table and the micropython filesystem. The layout table specifies the address range of the micropython filesystem, which is the area the PF code should work on.
MicroBitFileSystem::reformat assumes too much about the filesystem start address. MakeCode starts the MicroBitFileSystem at a different address.
I think having the PF code use MicroBitFileSystem is unnecessarily complicating the situation. Assuming that a PF should erase the MicroBitFileSystem is especially problematic if MicroBitFileSystem were to move into KL27 memory.

and int MicroBitFileSystem::format() states

Initialises a new file system. Assumes all pages are already erased.

I hadn't noticed that. I think it isn't true. Maybe it was in a early version?

However this has made me realise that 0xc71c0011 is for an all 0s page, not an all 0xFF page and will need to be changed

This didn't occur to me at first, but there are 4,294,967,296 possible checksums and 256^4096 combinations of bytes in a page. That's a very large number of combinations per checksum (>3e+9854). A simple scan for any non-FF would be safer and quicker.

I think micropython has it's own file system and does not use MicroBitFileSystem.

Oh sorry, you're right. I've always thought they were one and the same. I'll rework this
Thanks Martin

So why erase all those pages that didn't need to be touched before?

Is this the MBFS pages that I'm erasing (incorrectly, I've now realised)? I'll remove that and look at clearing the MicroPython filesystem now
If you mean the embedded source page, it should have previously only removed the embedded source magic, and nothing else. Can you link to the line you're concerned about as I'm not sure where you mean

The micropython table is at the end of the page containing FLASH_PROGRAM_END/mp_end or the next page if there's no room in that page.

I think once I change to erasing the correct filesystem, I'll remove the changes to MicroBitFilesystem to keep this PR tidy, and I'll bare this in mind when doing so

A simple scan for any non-FF would be safer and quicker.

True, I guess the CRC will have to iterate through each byte anyway so I might as well check them myself

I'm sorry I mentioned the MicroBitFileSystem! I had a feeling they were different, but hadn't tracked down the micropython code until today.

If you mean the embedded source page, it should have previously only removed the embedded source magic, and nothing else.
Yes, in the current master, this code overwrites the MakeCode source magic. Because it writes zero, it will not erase the page.
https://github.com/lancaster-university/codal-microbit-v2/blob/master/source/bluetooth/MicroBitPartialFlashingService.cpp#L333

Early on in this PR, I think this changed to erasing all pages after the PF data, which is required for MicroPython, short of parsing the MicroPython file system. After a quick look at the MicroPython file system code, it appears the page at the end starting FD will never be blank, because that's a page type byte, but that page doesn't have to be at the end of the region.
My instinct is to try not to, to avoid side effects, but it may be that erasing the PF area is the best simple solution, There are a lot of cases to think about, and it only potentially means one extra erase per page (and wiping the MicroBitFileSystem).
There's an instant in a MakeCode PF where the MakeCode magic is erased. That instant will be a while if all pages are erased before starting to write.
MicroBitMemoryMap can tell, but doesn't record or return if it's MakeCode or MicroPython.

My hex parser was ignoring the scratch page, fixed that and this works as expected

Hey @martinwork, please could you give this a quick look

I have added a few comments.

thanks @microbit-sam  @martinwork !


Automatically publish GitHub releases on tag push

Runs separate workflow only on tag pushes to publish release.
What it does:

Extracts current release changes from CHANGELOG.md (everything between ## [${CIRCLE_TAG#v}] and next version header in file)
Determines whether this is pre-release by checking whether there is any prefix in tag name (so 2.16.0 is a release, but 2.16.0-preview125 is a pre-release)
Calls github-release tool to make the release itself.

How to set up:

Generate new personal access token with public_repo scope and write it to CircleCI project settings' Environment variables section as GITHUB_TOKEN.
Merge this PR
Push version tags as usual

Examples:

Pre-release: https://github.com/Envek/imgproxy/releases/tag/v2.16.0-epsilon
Release: https://github.com/Envek/imgproxy/releases/tag/v2.16.0

Let's try it out :)

Dang! Works like a charm 👍


Where does the config go?

Hi there,
I know this might seem like a very stupid question that might have an extremely trivial answer, but where do I set the variables?
I simply can't seem to figure it out, as I've been messing around in my init.vim for the past 15 minutes to no avail.
Mind helping me out?

Apologies, I've figured it out!
Apprently coc has a command called CocConfig that'll let you edit a JSON file with configuration.
Sorry for the bother!

Apologies, I've figured it out!
Apprently coc has a command called CocConfig that'll let you edit a JSON file with configuration.
Sorry for the bother!


Add non-Gregorian calender conversions

[ ] add convertdate as an optional dependency
[ ] provide functions for computing at least the new year's days or the first day of each month in the respective calendar system
[ ] take care of varying calendar year lengths and include functions for the second appearance of a month in a Gregorian year

Since function discovery has been implemented using entry-points it would be more natural to implement this as a separate package.


"Point in direction" editor is buggy on Firefox

Actual Behavior
A bit buggy on Firefox Quantum 57.0.4 (32-bit) - when entered "point in direction" wheel, the input value stays and keeps changing input of other values when trying to edit.
Steps to Reproduce
(Assumed, needs repro)

Open the "point in direction" wheel
Start dragging the arrow
Release the arrow
Move the mouse around
See that the arrow keeps moving

Operating System and Browser
Firefox Quantum 57.0.4 on Windows

This bug was mentioned again in this recent forum post: https://scratch.mit.edu/discuss/topic/319343/.
The author didn't mention Firefox but that's the only browser that I found to have an issue.  It's fairly easy to make happen but this long set of steps is the only way I could reproduce it and capture it.  It's an annoying bug because once it occurs your editor session is in a limbo.
To reproduce:

Start the Firefox Browser (mine is at 63.0) under Windows 7

Go to beta.scratch.mit.edu, click Try It!

Drag a "Point in Direction" block to the scripting area.

and drag out another block with an input field (Such as "set my variable to" block)

Click on the 90 of the "Point in Direction" block.

I dragged (button down) the pointer to the -120 value.

lift mouse button and move it slightly to change to an arrow.  Press down the mouse button.

Move slightly until it turns back into a hand - but start over if you get the circle with a backslash.

At this point the white hand of the pointer and the blue arrow in the white circle should move with the mouse button up.

Stop at -120 and click (don't hold down) the mouse button.

Go the the input area of the set variable block.

Just clicking the field sets a value.
Move the mouse around and notice that the direction input value changes.
Windows 7; Firefox 63.0


EngineVGui Error

Unknown, Cant get it from not showing up.
Config used:
(https://files.catbox.moe/u39op9.fw)

Description:
Fedoraware won't open in tf2, and injecting fedoraware gives a error : EngineVGui.
Steps to reproduce:

[ Use TFGhost V2, Launch tfghost, Inject fedoraware. TF2 Will open a new small window saying EngineVGui, With the top name saying Clinterfaces::Init() -> nullptr. Opening the actual tf2 window and press insert wont open the GUI.] I AM able to consistently reproduce this error

Steps to reproduce the behavior:

Launch TFGhost
Inject Fedoraware
Look at the second TF2 Window

Additional context

TFGhost ia an anti VAC ban launcher, dont know if that helps.

[x] I have included the steps to reproduce to the best I am able to share.

Have you downloaded the latest build?

Thanks, didn't realize it updated.


[FEA] Rust API

Rust API has been highly requested by users. Ideally this would be a wrapper around the C-API

Closed with #34


Abstract ExecutionPathTracer into its own project

As part of future efforts to use the tracer in other projects, this PR takes the first step at decoupling ExecutionPathTracer from IQSharp.Core and moving it into its own project under the Tracer folder (using ExecutionPathTracer resulted in namespace/class name conflict).

As a quick note, the term "tracer" has often been used informally to refer to the trace simulator.

As a quick note, the term "tracer" has often been used informally to refer to the trace simulator.

Thanks @cgranade, I didn't realize that. Maybe then TraceVisualizer would be less ambiguous?

@rmshaffer @cgranade I did consider that as well, and I'm open to suggestions! I thought of TraceVisualizer but this folder doesn't do any of the visualization, just traces out the operations into an intermediate representation to pass to the Visualizer (JS).

@rmshaffer Ryan Shaffer FTE @cgranade Chris Granade FTE I did consider that as well, and I'm open to suggestions! I thought of TraceVisualizer but this folder doesn't do any of the visualization, just traces out the operations into an intermediate representation to pass to the Visualizer (JS).

That's a good point as well. Going back to your original comment, I wonder why ExecutionPathTracer doesn't work? e.g. in AzureClient project, both the namespace and a public class within the namespace are called AzureClient and everything seems to work out.

@rmshaffer I'll try it out again haha

@rmshaffer ah, it might have been solved in src\Kernel\Magic\Resolution\MagicResolver.cs L45 with AzureClient/AzureClient.


Building for arm64 device, arm64 simulator and x86_64 simulator

Following on from https://github.com/google/j2objc/issues/1828 ...
I'm not sure I was exactly clear, or @tomball maybe read in a hurry :) My problem isn't that I can't build for arm64 on M1. My problem is that building on an Intel Mac ONLY builds for arm64, for some reason.
For completeness, I followed your suggestion in the last ticket and executed:
J2OBJC_ARCHS="iphone64 simulator64 macosx64 maccatalyst64" make -j8 frameworks
And the resulting built frameworks appear to have arm64 binaries only:
rory@Eddie ~/W/j2objc (master)> ls dist/frameworks/JRE.xcframework/
Headers			Versions		ios-arm64-simulator
Info.plist		ios-arm64

This is weird, no? Why on earth am I only getting arm64 binaries for device and simulator? This also happens on my M1 machine, for reference.
Thanks!

Original ticket copy below:
We're in the midst of trying to update our j2objc library to XCFramework, and are hoping to be able to run our app natively on the arm64 Simulator on M1 Macs. I'm trying to build j2objc from source (latest master as of today - commit f119c4274dde21785dc8d96bef52f3dc3a15ecac).
I ran the following:
J2OBJC_ARCHS="iphone64 simulator64" make -j8 frameworks

However, no matter if i build on my M1 Air or my Intel iMac, this results in XCFrameworks with only arm64 binaries:
rory@Eddie ~/W/j/d/frameworks (master)> ls JRE.xcframework/
Headers			Versions		ios-arm64-simulator
Info.plist		ios-arm64
rory@Eddie ~/W/j/d/frameworks (master)> file JRE.xcframework/ios-arm64-simulator/libjre_emul.a
JRE.xcframework/ios-arm64-simulator/libjre_emul.a: Mach-O universal binary with 1 architecture: [arm64:current ar archive]
JRE.xcframework/ios-arm64-simulator/libjre_emul.a (for architecture arm64):	current ar archive
rory@Eddie ~/W/j/d/frameworks (master)> file JRE.xcframework/ios-arm64/libjre_emul.a
JRE.xcframework/ios-arm64/libjre_emul.a: Mach-O universal binary with 1 architecture: [arm64:current ar archive]
JRE.xcframework/ios-arm64/libjre_emul.a (for architecture arm64):	current ar archive
rory@Eddie ~/W/j/d/frameworks (master)>

I first did the build on my M1 so I thought it was just picking up on the local CPU arch, but I repeated on my intel iMac and got the same result.
Our own library build (including j2objc-converted Java code) generates an XCFramework with a fat-binary for Simulator:
rory@Hactar ~/W/l/b/LibASW-iOS.xcframework> ls
Info.plist			ios-arm64_x86_64-simulator
ios-arm64
... etc

though admittedly this is largely built 'by hand' in our build script.
Am I doing something wrong? My goal is to use JRE.xcframework in our app, along with our own library, and have it run natively on both Intel and Apple Silicon Macs.

Giving it a try with defaults, i.e. no explicit J2OBJC_ARCHS env var set... will report results shortly.

Ok, that took a long time, and eventually failed with:
building JRE framework
xcodebuild -create-xcframework -output ../dist/frameworks/JRE.xcframework -library /Users/rory/Workspace/j2objc/jre_emul/build_result/iphone/libjre_emul.a -library /Users/rory/Workspace/j2objc/jre_emul/build_result/simulator/libjre_emul.a -library /Users/rory/Workspace/j2objc/jre_emul/build_result/maccatalyst/libjre_emul.a -library /Users/rory/Workspace/j2objc/jre_emul/build_result/watchos/libjre_emul.a -library /Users/rory/Workspace/j2objc/jre_emul/build_result/watchsimulator/libjre_emul.a -library /Users/rory/Workspace/j2objc/jre_emul/build_result/appletvos/libjre_emul.a
error: binaries with multiple platforms are not supported '/Users/rory/Workspace/j2objc/jre_emul/build_result/appletvos/libjre_emul.a'
make[1]: *** [../dist/frameworks/JRE.xcframework] Error 70
make: *** [frameworks] Error 2

Ok, I just realised maybe im misunderstanding what simulator64 actually means. I thought it meant build for 64-bit platforms using the Simulator SDK. But does it actually exclusively mean build for arm64 with Simulator SDK?
Going to try a build with
J2OBJC_ARCHS="simulator simulator64 iphone64"

Yep! Problem solved, sorry for the misunderstanding.
Headers				Versions			ios-arm64_x86_64-simulator
Info.plist			ios-arm64

Will close now.

FYI: the failure building appletvos should be fixed in PR #1829, now being submitted.

Good to know, thanks @tomball