proposal: spec: add native support for programming persistent memory in Go

Persistent Memory is a new memory technology that allows byte-addressability at DRAM-like access speed and provides disk-like persistence. Applications using persistent memory benefit in a number of ways such as seeing improved performance and faster restart times. More details on this technology can be found at pmem.io.
This is a proposal to add native support for programming persistent memory in Go. A detailed design of our approach to add this support is described in our 2020 USENIX ATC paper go-pmem. An implementation of the above design based on Go 1.15 release is available here. In summary, adding support for natively programming persistent memory requires following capabilities to be added to Go :

Manage a garbage-collected persistent memory heap
Provide an interface for applications to allocate objects in persistent memory heap
Enable applications to make crash-consistent updates to data in persistent memory
Support applications to recover following a crash/restart

There exists libraries such as Intel PMDK that provides C and C++ developers support for persistent memory programming. Other programming languages such as Java and Python are also exploring ways to enable efficient access to persistent memory.  But no language provide a native persistent memory programming support. This proposal attempts to remedy this problem by making Go the first language to completely support persistent memory.
Since adding this support involves significant changes to the language runtime and compiler, we have also prepared a design document that I will attach to this proposal.

Design document - https://go-review.googlesource.com/c/proposal/+/284992
cc: @mohit10verma @pratapsu

Design document - https://go-review.googlesource.com/c/proposal/+/284992
cc: @mohit10verma @pratapsu

I spend enough time explaining to new Go users why they shouldn't use println. Please don't make me explain why they shouldn't use pmake.

I spend enough time explaining to new Go users why they shouldn't use println. Please don't make me explain why they shouldn't use pmake.

Because...

Honestly, I don't like the idea of persistent memory in general, but I especially don't like the idea of it in a GC language like Go. Please don't.

In case there is a pointer to non-persistent memory from persistent memory, all these pointers are zeroed out when a program restarts

How is that better than pointing to a random area of memory? I guess the program blows up deferences a nil pointer rather than a pointer to random memory, but this doesn’t sound very useful if every reference type is spring loaded with nil even if the original author went to lengths to avoid creating objects with invalid references.
Also, how does persistent memory keep track of everything that is a pointer without confusing it with large numbers, floats, strings, etc?

In case there is a pointer to non-persistent memory from persistent memory, all these pointers are zeroed out when a program restarts

How is that better than pointing to a random area of memory? I guess the program blows up deferences a nil pointer rather than a pointer to random memory, but this doesn’t sound very useful if every reference type is spring loaded with nil even if the original author went to lengths to avoid creating objects with invalid references.
Also, how does persistent memory keep track of everything that is a pointer without confusing it with large numbers, floats, strings, etc?

If the language supported generics, then as the design doc notes we could implement pnew and pmake as library functions. Can txn be implemented as a library function? That is, could we have beginTransaction and endTransaction? Or does it have to be implemented in the compiler?

Within a txn block, we instrument stores to persistent memory and inject calls to an undo log before the actual updates are made. (We have a SSA pass, similar to writeBarrier SSA pass to do this). These logs are needed to recover from crashes should they occur. Even with a library, the compiler needs to be aware of a transactional region and instrument the stores.
Alternatively, instead of a language change we can have pragmas like //go:transactional for functions.

If the language supported generics, then as the design doc notes we could implement pnew and pmake as library functions. Can txn be implemented as a library function? That is, could we have beginTransaction and endTransaction? Or does it have to be implemented in the compiler?

Within a txn block, we instrument stores to persistent memory and inject calls to an undo log before the actual updates are made. (We have a SSA pass, similar to writeBarrier SSA pass to do this). These logs are needed to recover from crashes should they occur. Even with a library, the compiler needs to be aware of a transactional region and instrument the stores.
Alternatively, instead of a language change we can have pragmas like //go:transactional for functions.

Hi Dave,
Thanks for the questions.

In case there is a pointer to non-persistent memory from persistent memory, all these pointers are zeroed out when a program restarts

How is that better than pointing to a random area of memory? I guess the program blows up deferences a nil pointer rather than a pointer to random memory, but this doesn’t sound very useful if every reference type is spring loaded with nil even if the original author went to lengths to avoid creating objects with invalid references.

Pointing to a random area of memory when an application restarts is much more dangerous than overwriting those pointers with nil pointers. The random area in memory could be an unmapped region of memory which upon accessing will result in the OS killing that application, or it could be a region in a span that is unused which violates the invariants kept by the garbage collector. Even worse, it could point to a valid region in memory, and the application can unitentionally corrupt such memory.
Even though zeroing is not ideal, we felt it is the right compromise we could arrive at. Application that incorrectly store such references will crash on accessing such data. Theoretically, it also facilitates a defensive style of programming where applications can check for the validity of each pointer in persistent memory before accessing such pointers.

Also, how does persistent memory keep track of everything that is a pointer without confusing it with large numbers, floats, strings, etc?

During the regular run of an application, persistent memory heap is managed by the Go runtime exactly like the volatile memory heap. They are managed in 64MB arenas, and the arena stores metadata about that region such as its heap type bitmap, span map, etc (these metadata are stored in volatile memory). In addition to this, we added code to log some of this metadata in a header region of each persistent memory arena. Two types of metadata are logged - heap type bitmaps and span table. Heap type bitmap stores the exact bitmap used by the Go runtime that is used to identify pointers in the heap. The span table stores information about each span in the arena such as its sizeclass, needzero parameter, number of pages (for large spans), etc. On an application restart, the heap type bitmap is restored as such from the persistent memory arena header to its corresponding arena metadata in volatile memory. Spans are recreated using the span table. The GC then uses the restored heap type bits to identify pointers in the heap (just like for volatile memory).

Hi Dave,
Thanks for the questions.

In case there is a pointer to non-persistent memory from persistent memory, all these pointers are zeroed out when a program restarts

How is that better than pointing to a random area of memory? I guess the program blows up deferences a nil pointer rather than a pointer to random memory, but this doesn’t sound very useful if every reference type is spring loaded with nil even if the original author went to lengths to avoid creating objects with invalid references.

Pointing to a random area of memory when an application restarts is much more dangerous than overwriting those pointers with nil pointers. The random area in memory could be an unmapped region of memory which upon accessing will result in the OS killing that application, or it could be a region in a span that is unused which violates the invariants kept by the garbage collector. Even worse, it could point to a valid region in memory, and the application can unitentionally corrupt such memory.
Even though zeroing is not ideal, we felt it is the right compromise we could arrive at. Application that incorrectly store such references will crash on accessing such data. Theoretically, it also facilitates a defensive style of programming where applications can check for the validity of each pointer in persistent memory before accessing such pointers.

Also, how does persistent memory keep track of everything that is a pointer without confusing it with large numbers, floats, strings, etc?

During the regular run of an application, persistent memory heap is managed by the Go runtime exactly like the volatile memory heap. They are managed in 64MB arenas, and the arena stores metadata about that region such as its heap type bitmap, span map, etc (these metadata are stored in volatile memory). In addition to this, we added code to log some of this metadata in a header region of each persistent memory arena. Two types of metadata are logged - heap type bitmaps and span table. Heap type bitmap stores the exact bitmap used by the Go runtime that is used to identify pointers in the heap. The span table stores information about each span in the arena such as its sizeclass, needzero parameter, number of pages (for large spans), etc. On an application restart, the heap type bitmap is restored as such from the persistent memory arena header to its corresponding arena metadata in volatile memory. Spans are recreated using the span table. The GC then uses the restored heap type bits to identify pointers in the heap (just like for volatile memory).

I spend enough time explaining to new Go users why they shouldn't use println. Please don't make me explain why they shouldn't use pmake.

Not sure what is the concern you are expressing here. Language changes such as pnew/pmake are not necessary to add support for programming persistent memory. We have explored an alternative using generics where these functionalities can be provided by functions exported by a package - https://github.com/golang/proposal/blob/master/design/43810-go-pmem.md#compatibility.

I spend enough time explaining to new Go users why they shouldn't use println. Please don't make me explain why they shouldn't use pmake.

Not sure what is the concern you are expressing here. Language changes such as pnew/pmake are not necessary to add support for programming persistent memory. We have explored an alternative using generics where these functionalities can be provided by functions exported by a package - https://github.com/golang/proposal/blob/master/design/43810-go-pmem.md#compatibility.

My concern is that pmem is not available on most hardware that developers are using today. That leads me to the question, whether it could be emulated with memory-mapped disk files?
Usually the Go philosophy is to implement abstract concepts and map them to hardware features if they are supported. Examples are goroutines or the functions in math/bits.

My concern is that pmem is not available on most hardware that developers are using today. That leads me to the question, whether it could be emulated with memory-mapped disk files?
Usually the Go philosophy is to implement abstract concepts and map them to hardware features if they are supported. Examples are goroutines or the functions in math/bits.

support this proposal

support this proposal

I have questions :)
Disclaimer: I'm not previously familiar with the concept of persistent memory, so some questions might well be misguided.
The docs say that pmemory is allocated similarly to Go, with the pnew function. But in Go the new builtin is almost never used. Instead, memory is most often allocated by using struct or slice literals. Would that be possible with this proposal? Can you allocate a map inside pmem? What about other kinds of data structures? I'm thinking that it would be nice to be able to re-use existing Go algorithm implementations without making a specialised copy that works only on pmem.
When you have a program that's using pmem you have two kinds of memory, not distinguished in the type system: volatile and persistent. If I understand correctly, if you accidentally store a pointer to volatile memory inside persistent memory, you'll lose that data on restart. I could see that as being a fruitful source of bugs: is this issue amenable to some kind of static analysis to avoid those bugs?
Persistent memory is accessed via a file-like API, so it should be possible to access more than one at a time, but the pmem package proposed here is global - it doesn't seem to provide any way to access more than one pmem file at a time. Is this a fundamental constraint?
Is pmem access fundamentally exclusive? That is, is it possible to run two instances of the same pmem program concurrently? In my brief scan through some documents and code, I didn't see anything that indicated that it was. And ISTM that it would potentially be useful to be able to have multiple processes running with access to the same persistent dataset. If access is not exclusive, that raises the issue of synchronisation primitives: would a sync.Mutex or a sync.Cond work in persistent memory? If access is not exclusive, then is it possible to do garbage collection at all? The possibility of having pointers from persistent to volatile memory seems particularly hazardous in this scenario.

I have questions :)
Disclaimer: I'm not previously familiar with the concept of persistent memory, so some questions might well be misguided.
The docs say that pmemory is allocated similarly to Go, with the pnew function. But in Go the new builtin is almost never used. Instead, memory is most often allocated by using struct or slice literals. Would that be possible with this proposal? Can you allocate a map inside pmem? What about other kinds of data structures? I'm thinking that it would be nice to be able to re-use existing Go algorithm implementations without making a specialised copy that works only on pmem.
When you have a program that's using pmem you have two kinds of memory, not distinguished in the type system: volatile and persistent. If I understand correctly, if you accidentally store a pointer to volatile memory inside persistent memory, you'll lose that data on restart. I could see that as being a fruitful source of bugs: is this issue amenable to some kind of static analysis to avoid those bugs?
Persistent memory is accessed via a file-like API, so it should be possible to access more than one at a time, but the pmem package proposed here is global - it doesn't seem to provide any way to access more than one pmem file at a time. Is this a fundamental constraint?
Is pmem access fundamentally exclusive? That is, is it possible to run two instances of the same pmem program concurrently? In my brief scan through some documents and code, I didn't see anything that indicated that it was. And ISTM that it would potentially be useful to be able to have multiple processes running with access to the same persistent dataset. If access is not exclusive, that raises the issue of synchronisation primitives: would a sync.Mutex or a sync.Cond work in persistent memory? If access is not exclusive, then is it possible to do garbage collection at all? The possibility of having pointers from persistent to volatile memory seems particularly hazardous in this scenario.

Since the tech is quite new, how's the security? I didn't find any papers evaluating security of it. Security is hardly mentioned in the docbook. From that perspective it's another Row Hammer / Spectre / etc waiting to happen.

Since the tech is quite new, how's the security? I didn't find any papers evaluating security of it. Security is hardly mentioned in the docbook. From that perspective it's another Row Hammer / Spectre / etc waiting to happen.

My concern is that pmem is not available on most hardware that developers are using today. That leads me to the question, whether it could be emulated with memory-mapped disk files?
Usually the Go philosophy is to implement abstract concepts and map them to hardware features if they are supported. Examples are goroutines or the functions in math/bits.

Operating systems like Linux can mmap persistent memory into the virtual address space. Using this, it is possible to fake "persistent memory" for testing purposes using something like RAMdisk etc.

My concern is that pmem is not available on most hardware that developers are using today. That leads me to the question, whether it could be emulated with memory-mapped disk files?
Usually the Go philosophy is to implement abstract concepts and map them to hardware features if they are supported. Examples are goroutines or the functions in math/bits.

Operating systems like Linux can mmap persistent memory into the virtual address space. Using this, it is possible to fake "persistent memory" for testing purposes using something like RAMdisk etc.

This seems to propose adding software transactional semantics to the base layer of Go. As far as I know, trying to create a practical STM has been an unremitting failure in every language that does not have type-system level support for isolating IO from STM, despite very significant effort poured into the effort by well-funded attempts. I see nothing about Go that would mitigate this in Go; quite the contrary, it lacks even some of the tools that languages like C# had, which still proved ultimately inadequate to the task.
Moreover, I envision trying to use this in practical code, and it seems like this API will ultimately prove very inadequate and become a real thorn-in-the-side on an ongoing basis as it requires more and more extensions (even beyond the STM problems in the previous paragraph) to make it even possible to address problems arising in field usage. For instance, the paper mentions that memory leaks are more dangerous in persisted memory because they survive a reboot. However... memory leaks will happen. "Just don't leak" isn't a solution. So, how are we supposed to address them? The mentioned API has nothing that could solve that problem that I can see. The obvious extensions will be nearly unusable, IMHO; it would take something relatively deep.
This would seem to be setting up for very long term support for this use case that will require wildly disproportional effort to maintain vs. the amount of use its going to have in the broader Go community compared to the existing code base.
It seems like you'd be much happier just forking Go and being able to do what you need to do to deepen this integration without having to run everything past the core Go team, who are always going to be having to balance your niche desires vs the much larger core and generally you're going to lose.

This seems to propose adding software transactional semantics to the base layer of Go. As far as I know, trying to create a practical STM has been an unremitting failure in every language that does not have type-system level support for isolating IO from STM, despite very significant effort poured into the effort by well-funded attempts. I see nothing about Go that would mitigate this in Go; quite the contrary, it lacks even some of the tools that languages like C# had, which still proved ultimately inadequate to the task.
Moreover, I envision trying to use this in practical code, and it seems like this API will ultimately prove very inadequate and become a real thorn-in-the-side on an ongoing basis as it requires more and more extensions (even beyond the STM problems in the previous paragraph) to make it even possible to address problems arising in field usage. For instance, the paper mentions that memory leaks are more dangerous in persisted memory because they survive a reboot. However... memory leaks will happen. "Just don't leak" isn't a solution. So, how are we supposed to address them? The mentioned API has nothing that could solve that problem that I can see. The obvious extensions will be nearly unusable, IMHO; it would take something relatively deep.
This would seem to be setting up for very long term support for this use case that will require wildly disproportional effort to maintain vs. the amount of use its going to have in the broader Go community compared to the existing code base.
It seems like you'd be much happier just forking Go and being able to do what you need to do to deepen this integration without having to run everything past the core Go team, who are always going to be having to balance your niche desires vs the much larger core and generally you're going to lose.

@mohit10verma Apologees for not being clear enough. I intended to ask whether persistent memory can be emulated on hardware without persistent memory. My guess is yes, but it would be much slower.

@mohit10verma Apologees for not being clear enough. I intended to ask whether persistent memory can be emulated on hardware without persistent memory. My guess is yes, but it would be much slower.

This seems to propose adding software transactional semantics to the base layer of Go. As far as I know, trying to create a practical STM has been an unremitting failure in every language that does not have type-system level support for isolating IO from STM, despite very significant effort poured into the effort by well-funded attempts. I see nothing about Go that would mitigate this in Go; quite the contrary, it lacks even some of the tools that languages like C# had, which still proved ultimately inadequate to the task.

Yes you are right. Making updates to persistent memory in a crash-consistent way with the help of language support needs Software Transactional Memory support. And as you mentioned there have been many unsuccessful efforts. For example C# tried and failed to support STM. Our current design also has open issues like how to handle IO within a transaction.
An alternative is to not have transactions in-built to Go, but leave it to the user to write their own transactions (like Databases do). This makes the programming difficult (all stores will have to be manually logged by the user. Our transaction package is an example of such a user-level package).
In this approach, Go manages the persistent memory heap (allocation + Garbage collection) and leaves it to the user to write transaction packages. Infact, we have seen this proposal for Java.

Moreover, I envision trying to use this in practical code, and it seems like this API will ultimately prove very inadequate and become a real thorn-in-the-side on an ongoing basis as it requires more and more extensions (even beyond the STM problems in the previous paragraph) to make it even possible to address problems arising in field usage. For instance, the paper mentions that memory leaks are more dangerous in persisted memory because they survive a reboot. However... memory leaks will happen. "Just don't leak" isn't a solution. So, how are we supposed to address them? The mentioned API has nothing that could solve that problem that I can see. The obvious extensions will be nearly unusable, IMHO; it would take something relatively deep.

“memory leaks will happen” is not correct.

This would seem to be setting up for very long term support for this use case that will require wildly disproportional effort to maintain vs. the amount of use its going to have in the broader Go community compared to the existing code base.

Yes it is long term and so we opened this proposal for the community to talk about.

It seems like you'd be much happier just forking Go and being able to do what you need to do to deepen this integration without having to run everything past the core Go team, who are always going to be having to balance your niche desires vs the much larger core and generally you're going to lose.

As you suggested, we do have a fork of Go-1.15 with all our changes. We maintain this fork here. Here is the design document we posted along with this proposal. Applications using persistent memory benefit from very fast restart times and more throughput compared to running on traditional IO devices. We already see some language communities discussing persistent memory (Java, Python extensions) and so we opened this proposal.

This seems to propose adding software transactional semantics to the base layer of Go. As far as I know, trying to create a practical STM has been an unremitting failure in every language that does not have type-system level support for isolating IO from STM, despite very significant effort poured into the effort by well-funded attempts. I see nothing about Go that would mitigate this in Go; quite the contrary, it lacks even some of the tools that languages like C# had, which still proved ultimately inadequate to the task.

Yes you are right. Making updates to persistent memory in a crash-consistent way with the help of language support needs Software Transactional Memory support. And as you mentioned there have been many unsuccessful efforts. For example C# tried and failed to support STM. Our current design also has open issues like how to handle IO within a transaction.
An alternative is to not have transactions in-built to Go, but leave it to the user to write their own transactions (like Databases do). This makes the programming difficult (all stores will have to be manually logged by the user. Our transaction package is an example of such a user-level package).
In this approach, Go manages the persistent memory heap (allocation + Garbage collection) and leaves it to the user to write transaction packages. Infact, we have seen this proposal for Java.

Moreover, I envision trying to use this in practical code, and it seems like this API will ultimately prove very inadequate and become a real thorn-in-the-side on an ongoing basis as it requires more and more extensions (even beyond the STM problems in the previous paragraph) to make it even possible to address problems arising in field usage. For instance, the paper mentions that memory leaks are more dangerous in persisted memory because they survive a reboot. However... memory leaks will happen. "Just don't leak" isn't a solution. So, how are we supposed to address them? The mentioned API has nothing that could solve that problem that I can see. The obvious extensions will be nearly unusable, IMHO; it would take something relatively deep.

“memory leaks will happen” is not correct.

This would seem to be setting up for very long term support for this use case that will require wildly disproportional effort to maintain vs. the amount of use its going to have in the broader Go community compared to the existing code base.

Yes it is long term and so we opened this proposal for the community to talk about.

It seems like you'd be much happier just forking Go and being able to do what you need to do to deepen this integration without having to run everything past the core Go team, who are always going to be having to balance your niche desires vs the much larger core and generally you're going to lose.

As you suggested, we do have a fork of Go-1.15 with all our changes. We maintain this fork here. Here is the design document we posted along with this proposal. Applications using persistent memory benefit from very fast restart times and more throughput compared to running on traditional IO devices. We already see some language communities discussing persistent memory (Java, Python extensions) and so we opened this proposal.

I have questions :)
Disclaimer: I'm not previously familiar with the concept of persistent memory, so some questions might well be misguided.

We do appreciate all the questions :)

The docs say that pmemory is allocated similarly to Go, with the pnew function. But in Go the new builtin is almost never used. Instead, memory is most often allocated by using struct or slice literals. Would that be possible with this proposal? Can you allocate a map inside pmem? What about other kinds of data structures? I'm thinking that it would be nice to be able to re-use existing Go algorithm implementations without making a specialised copy that works only on pmem.

Our implementation do not support allocating persistent memory using struct or slice literals. This is not a fundamental limitation - just that we have not added such a support. But it will definitely need some new syntax to distinguish between persistent and volatile memory allocations.
We do not support creating maps in persistent memory using our pmake() API. Other datastructures can be allocated using the pnew() API. To an application, persistent memory looks like just another region in the heap, and data in persistent memory are accessed using regular pointers. So existing Go algorithms can run on data in persistent memory but they may not offer important guarantees such as crash consistency.

When you have a program that's using pmem you have two kinds of memory, not distinguished in the type system: volatile and persistent. If I understand correctly, if you accidentally store a pointer to volatile memory inside persistent memory, you'll lose that data on restart. I could see that as being a fruitful source of bugs: is this issue amenable to some kind of static analysis to avoid those bugs?

Yes, we do allow pointers to volatile memory to be store inside persistent memory. On application restart, any such pointers are zeroed out. We do agree this is not ideal. Static analysis is unlikely to catch all sources of such bugs as what gets stored in a pointer is very run-time dependent. We have seen few solutions being proposed to mitigate such problems. For example, Autopersist uses a reachability analysis to automatically persist all objects reachable from a durable root in persistent memory.

Persistent memory is accessed via a file-like API, so it should be possible to access more than one at a time, but the pmem package proposed here is global - it doesn't seem to provide any way to access more than one pmem file at a time from the same program. Is this a fundamental constraint?

This is not a fundamental constraint. It is possible to extend the API to allow persistent memory to be mapped from multiple files. The PMDK library from Intel, which is a persistent memory programming library for C and C++, provides a similar functionalities - allowing users to create multiple pools, where each pool is backed by a separate file in persistent memory. But PMDK has the limitation that each pool is created of a fixed size.  In our design, applications start with a small heap that is dynamically expanded on demand. Multiple objects groups can be identied using named objects. So we did not find a need to have persistent memory be mapped from multiple files.

Is pmem access fundamentally exclusive? That is, is it possible to run two instances of the same pmem program concurrently? In my brief scan through some documents and code, I didn't see anything that indicated that it was. And ISTM that it would potentially be useful to be able to have multiple processes running with access to the same persistent dataset. If access is not exclusive, that raises the issue of synchronisation primitives: would a sync.Mutex or a sync.Cond work in persistent memory? If access is not exclusive, then is it possible to do garbage collection at all? The possibility of having pointers from persistent to volatile memory seems particularly hazardous in this scenario.

Our model exposes each persistent memory file to be used exclusively by a process - i.e., we assume another process is not using the same file at the same time. But again, this is not a fundamental limitation. Persistent memory files are like regular files on a disk. It is possible for two processes to map the same file into their address space and carefully curate acceses to the same file but in our experience this is very unusual. sync.Mutex and sync.Cond do work in persistent memory, but they may need to be reinitialized during application restart to ensure no stale state persists.

I have questions :)
Disclaimer: I'm not previously familiar with the concept of persistent memory, so some questions might well be misguided.

We do appreciate all the questions :)

The docs say that pmemory is allocated similarly to Go, with the pnew function. But in Go the new builtin is almost never used. Instead, memory is most often allocated by using struct or slice literals. Would that be possible with this proposal? Can you allocate a map inside pmem? What about other kinds of data structures? I'm thinking that it would be nice to be able to re-use existing Go algorithm implementations without making a specialised copy that works only on pmem.

Our implementation do not support allocating persistent memory using struct or slice literals. This is not a fundamental limitation - just that we have not added such a support. But it will definitely need some new syntax to distinguish between persistent and volatile memory allocations.
We do not support creating maps in persistent memory using our pmake() API. Other datastructures can be allocated using the pnew() API. To an application, persistent memory looks like just another region in the heap, and data in persistent memory are accessed using regular pointers. So existing Go algorithms can run on data in persistent memory but they may not offer important guarantees such as crash consistency.

When you have a program that's using pmem you have two kinds of memory, not distinguished in the type system: volatile and persistent. If I understand correctly, if you accidentally store a pointer to volatile memory inside persistent memory, you'll lose that data on restart. I could see that as being a fruitful source of bugs: is this issue amenable to some kind of static analysis to avoid those bugs?

Yes, we do allow pointers to volatile memory to be store inside persistent memory. On application restart, any such pointers are zeroed out. We do agree this is not ideal. Static analysis is unlikely to catch all sources of such bugs as what gets stored in a pointer is very run-time dependent. We have seen few solutions being proposed to mitigate such problems. For example, Autopersist uses a reachability analysis to automatically persist all objects reachable from a durable root in persistent memory.

Persistent memory is accessed via a file-like API, so it should be possible to access more than one at a time, but the pmem package proposed here is global - it doesn't seem to provide any way to access more than one pmem file at a time from the same program. Is this a fundamental constraint?

This is not a fundamental constraint. It is possible to extend the API to allow persistent memory to be mapped from multiple files. The PMDK library from Intel, which is a persistent memory programming library for C and C++, provides a similar functionalities - allowing users to create multiple pools, where each pool is backed by a separate file in persistent memory. But PMDK has the limitation that each pool is created of a fixed size.  In our design, applications start with a small heap that is dynamically expanded on demand. Multiple objects groups can be identied using named objects. So we did not find a need to have persistent memory be mapped from multiple files.

Is pmem access fundamentally exclusive? That is, is it possible to run two instances of the same pmem program concurrently? In my brief scan through some documents and code, I didn't see anything that indicated that it was. And ISTM that it would potentially be useful to be able to have multiple processes running with access to the same persistent dataset. If access is not exclusive, that raises the issue of synchronisation primitives: would a sync.Mutex or a sync.Cond work in persistent memory? If access is not exclusive, then is it possible to do garbage collection at all? The possibility of having pointers from persistent to volatile memory seems particularly hazardous in this scenario.

Our model exposes each persistent memory file to be used exclusively by a process - i.e., we assume another process is not using the same file at the same time. But again, this is not a fundamental limitation. Persistent memory files are like regular files on a disk. It is possible for two processes to map the same file into their address space and carefully curate acceses to the same file but in our experience this is very unusual. sync.Mutex and sync.Cond do work in persistent memory, but they may need to be reinitialized during application restart to ensure no stale state persists.

Since the tech is quite new, how's the security? I didn't find any papers evaluating security of it. Security is hardly mentioned in the docbook. From that perspective it's another Row Hammer / Spectre / etc waiting to happen.

I am not a security expert but I found some references to the security model of persistent memory.
This document mentions that all the rest data on persistent memory is encrypted using 256-bit AES hardware encryption.

Since the tech is quite new, how's the security? I didn't find any papers evaluating security of it. Security is hardly mentioned in the docbook. From that perspective it's another Row Hammer / Spectre / etc waiting to happen.

I am not a security expert but I found some references to the security model of persistent memory.
This document mentions that all the rest data on persistent memory is encrypted using 256-bit AES hardware encryption.

@mohit10verma Apologees for not being clear enough. I intended to ask whether persistent memory can be emulated on hardware without persistent memory. My guess is yes, but it would be much slower.

Hi  @ulikunitz,
Our programming model requires users to provide a file as an argument to the pmem.Init() API that initializes persistent memory. But this file need not reside in persistent memory. It could be a file in a ramdisk as Mohit mentioned, or a file on a regular disk. These are great to experiment with the new programming model for persistent memory but they will not offer the data persistence or data access speed benefits (respectively) of using persistent memory.
It is also possible to configure Linux boot options to earmark a portion of the DRAM to appear as persistent memory. See this guide for an example. Emulating DRAM as persistent memory gives you DRAM-like access speed but no data persistence. Hope that clarifies.

@mohit10verma Apologees for not being clear enough. I intended to ask whether persistent memory can be emulated on hardware without persistent memory. My guess is yes, but it would be much slower.

Hi  @ulikunitz,
Our programming model requires users to provide a file as an argument to the pmem.Init() API that initializes persistent memory. But this file need not reside in persistent memory. It could be a file in a ramdisk as Mohit mentioned, or a file on a regular disk. These are great to experiment with the new programming model for persistent memory but they will not offer the data persistence or data access speed benefits (respectively) of using persistent memory.
It is also possible to configure Linux boot options to earmark a portion of the DRAM to appear as persistent memory. See this guide for an example. Emulating DRAM as persistent memory gives you DRAM-like access speed but no data persistence. Hope that clarifies.

@jerrinsg Thanks, excellent explanation.

@jerrinsg Thanks, excellent explanation.

Our implementation do not support allocating persistent memory using struct or slice literals. This is not a fundamental limitation - just that we have not added such a support. But it will definitely need some new syntax to distinguish between persistent and volatile memory allocations.

Yes, we do allow pointers to volatile memory to be store inside persistent memory. On application restart, any such pointers are zeroed out. We do agree this is not ideal.

This is not a fundamental constraint. It is possible to extend the API to allow persistent memory to be mapped from multiple files.

Our model exposes each persistent memory file to be used exclusively by a process - i.e., we assume another process is not using the same file at the same time. But again, this is not a fundamental limitation.

ISTM that there are enough open possibilities here that it would be wrong to tie Go down by accepting a proposal at this stage. It's very early days for persistent memory and we still don't know how things might develop.
Persistent memory is definitely an interesting area to explore, but I think this issue is a non-starter at the current time and in its current state.

Our implementation do not support allocating persistent memory using struct or slice literals. This is not a fundamental limitation - just that we have not added such a support. But it will definitely need some new syntax to distinguish between persistent and volatile memory allocations.

Yes, we do allow pointers to volatile memory to be store inside persistent memory. On application restart, any such pointers are zeroed out. We do agree this is not ideal.

This is not a fundamental constraint. It is possible to extend the API to allow persistent memory to be mapped from multiple files.

Our model exposes each persistent memory file to be used exclusively by a process - i.e., we assume another process is not using the same file at the same time. But again, this is not a fundamental limitation.

ISTM that there are enough open possibilities here that it would be wrong to tie Go down by accepting a proposal at this stage. It's very early days for persistent memory and we still don't know how things might develop.
Persistent memory is definitely an interesting area to explore, but I think this issue is a non-starter at the current time and in its current state.

Our current design also has open issues like how to handle IO within a transaction.

This is the issue. Adding the primitives was never the problem for other languages. In practice, as you compose transactions together the probability that something would go wrong with repeating IO approached 1 too quickly for real programmers. What works in a small sample function didn't work in the real world.
The reason I suggest you just fork Go is that with that freedom, you stand a chance of solving it. I can imagine three possibilities to explore off the top of my head.

“memory leaks will happen” is not correct.

I've written plenty of Go programs that leak, obviously not GC, but because I appended all my log messages to a slice and never cleared it or whatever. I can solve it with a restart in current Go. Persistent memory is going to need other solutions to this, because real code is going to do things like create new named objects all the time without properly keeping track of them (even if they are nominally "live" by GC standards), thus "leaking". This is going to be a large problem in practice.
And this is a great example of what I mean by the freedom to explore. You're going to find out you need solutions. Your first crack is, with all due respect, likely to not be as good as it could be. You need the freedom to change your answers at this phase.
You have a paper and a demo. That is good work and true progress; do not let me sound like I think otherwise. But the next step now is to publicize your demo and try to create a community around it in the persistent memory community, to get some prototyping going and find out what works and what doesn't in some more real programs, with real programmers. You're on step one and trying to skip to step twelve.
I do not mean this as discouragement... I mean this post as encouragement, and license, and permission! Fork the code, explore, learn, don't wait for approval! "Let's start from a working general-purpose language and fix it to work really well with persistent memory!" is a solid value proposition to create a community around, because of the Go implementation's relative simplicity compared to almost any other language of comparable capability. This competitive time in the technology's life is no time to be running everything past a committee and, well, naysayers like me! (Though I'm actually trying to be a "yea-sayer" here, it just may not be the "yea" you may have been looking for. I cheer you on and hope you create something wonderful!)
Explore! Create! Have some fun!

Our current design also has open issues like how to handle IO within a transaction.

This is the issue. Adding the primitives was never the problem for other languages. In practice, as you compose transactions together the probability that something would go wrong with repeating IO approached 1 too quickly for real programmers. What works in a small sample function didn't work in the real world.
The reason I suggest you just fork Go is that with that freedom, you stand a chance of solving it. I can imagine three possibilities to explore off the top of my head.

“memory leaks will happen” is not correct.

I've written plenty of Go programs that leak, obviously not GC, but because I appended all my log messages to a slice and never cleared it or whatever. I can solve it with a restart in current Go. Persistent memory is going to need other solutions to this, because real code is going to do things like create new named objects all the time without properly keeping track of them (even if they are nominally "live" by GC standards), thus "leaking". This is going to be a large problem in practice.
And this is a great example of what I mean by the freedom to explore. You're going to find out you need solutions. Your first crack is, with all due respect, likely to not be as good as it could be. You need the freedom to change your answers at this phase.
You have a paper and a demo. That is good work and true progress; do not let me sound like I think otherwise. But the next step now is to publicize your demo and try to create a community around it in the persistent memory community, to get some prototyping going and find out what works and what doesn't in some more real programs, with real programmers. You're on step one and trying to skip to step twelve.
I do not mean this as discouragement... I mean this post as encouragement, and license, and permission! Fork the code, explore, learn, don't wait for approval! "Let's start from a working general-purpose language and fix it to work really well with persistent memory!" is a solid value proposition to create a community around, because of the Go implementation's relative simplicity compared to almost any other language of comparable capability. This competitive time in the technology's life is no time to be running everything past a committee and, well, naysayers like me! (Though I'm actually trying to be a "yea-sayer" here, it just may not be the "yea" you may have been looking for. I cheer you on and hope you create something wonderful!)
Explore! Create! Have some fun!

Since the tech is quite new, how's the security?

Persistent memory is accessed similar to DRAM with load/store instructions, different levels of caches, etc. So it inherits any known cache side channel attacks. I am not a device expert, but I don't think Row_Hammer immediately applies to persistent memory because Row_Hammer depends on how DRAM device works.
As memory encryption technologies like TME and MKTME 1 mature, I believe they will work for persistent memory too. In addition, as Mohit points out, current persistent memory devices have capability to encrypt/decrypt data at store/load.

Since the tech is quite new, how's the security?

Persistent memory is accessed similar to DRAM with load/store instructions, different levels of caches, etc. So it inherits any known cache side channel attacks. I am not a device expert, but I don't think Row_Hammer immediately applies to persistent memory because Row_Hammer depends on how DRAM device works.
As memory encryption technologies like TME and MKTME 1 mature, I believe they will work for persistent memory too. In addition, as Mohit points out, current persistent memory devices have capability to encrypt/decrypt data at store/load.

My primary concern about this proposal is the goal of "making Go the first language to completely support persistent memory". Being the first means making lots of mistakes and spending a lot of time redesigning, cleaning them up, and so on. I was once asked about the impact of the research literature on Go's design, and I replied in part:

Go is more an engineering project than a pure research project. Like most engineering, it is fundamentally conservative, using ideas that are proven and well understood and will work well together.

We really can't say that this approach to persistent memory is proven, nor well understood, nor that it will work well together with the other ideas in Go.
The cost of adopting a design that turns out not to be right is high. We have to keep supporting it even though it's not right, for backwards compatibility. On top of that, this particular design requires significant work in the garbage collector - a critical part of Go - and would limit future changes there. We really would want to be sure that we're not taking on a net cost as opposed to a net benefit, and I don't see the evidence for that.
Part of the reason we spent so long on generics was to avoid making mistakes. I think we've ended up in a decent place there, far better than we would have if we'd run with the best ideas we had in, say, 2012. We wanted to get something we'd be happy supporting for the next 10, 20 years. Waiting for the right design is a good strategy, one we have employed repeatedly.
I don't mean to discount the potentially transformative impact that support for and widespread availability of persistent memory might bring to programming. But this proposal is asking us to take a very big step on something and to lock in a particular way of doing things that may or may not be the right approach even a couple years from now, much less 10 or more.
I wonder whether there is any kind of generalization we could do that would let us make a much smaller change to the language and enable much more of the persistent memory "runtime" to live as an importable package maintained outside the core Go system.

My primary concern about this proposal is the goal of "making Go the first language to completely support persistent memory". Being the first means making lots of mistakes and spending a lot of time redesigning, cleaning them up, and so on. I was once asked about the impact of the research literature on Go's design, and I replied in part:

Go is more an engineering project than a pure research project. Like most engineering, it is fundamentally conservative, using ideas that are proven and well understood and will work well together.

We really can't say that this approach to persistent memory is proven, nor well understood, nor that it will work well together with the other ideas in Go.
The cost of adopting a design that turns out not to be right is high. We have to keep supporting it even though it's not right, for backwards compatibility. On top of that, this particular design requires significant work in the garbage collector - a critical part of Go - and would limit future changes there. We really would want to be sure that we're not taking on a net cost as opposed to a net benefit, and I don't see the evidence for that.
Part of the reason we spent so long on generics was to avoid making mistakes. I think we've ended up in a decent place there, far better than we would have if we'd run with the best ideas we had in, say, 2012. We wanted to get something we'd be happy supporting for the next 10, 20 years. Waiting for the right design is a good strategy, one we have employed repeatedly.
I don't mean to discount the potentially transformative impact that support for and widespread availability of persistent memory might bring to programming. But this proposal is asking us to take a very big step on something and to lock in a particular way of doing things that may or may not be the right approach even a couple years from now, much less 10 or more.
I wonder whether there is any kind of generalization we could do that would let us make a much smaller change to the language and enable much more of the persistent memory "runtime" to live as an importable package maintained outside the core Go system.

Thanks Russ for your response. We understand that our design is not well-proven and there could be flaws that we have not thought about. Even if our proposal is not acccepted in its entirety, we feel it still can be broken into pieces that can be used to add support for programming persistent memory in a phased manner.
The first step can be to provide an interface which lets a Go program map a persistent memory file into its address space. One option could be to support an additional flag MAP_SYNC [2] in the Mmap API to let users map files that specifically support direct-access (DAX). Also, data written to persistent memory can be guaranteed to be made persistent only when they are flushed from the CPU caches as well. An API needs to be made available that can flush a range of memory address using appropriate CPU instructions.
Adding these two support will make it possible for Go programs to easily program persistent memory. It then becomes possible for external packages to be written which provides other features such as memory management, transactions, etc. Note that this is similar to what was implemented in OpenJDK through JEP 352 [1]
There are advantages to providing these features in the core runtime than as an external package - these are the bare-minimal changes required to support persistent memory programming and it becomes much easier later to expand the language to add additional features such as a garbage-collected persistent memory heap, support for transactionally updating persistent memory datastructures, etc. If there is sufficient interest to expand on the initial support added to Go, more features can be added later on.
[1] https://bugs.openjdk.java.net/browse/JDK-8207851
[2] https://man7.org/linux/man-pages/man2/mmap.2.html

Thanks Russ for your response. We understand that our design is not well-proven and there could be flaws that we have not thought about. Even if our proposal is not acccepted in its entirety, we feel it still can be broken into pieces that can be used to add support for programming persistent memory in a phased manner.
The first step can be to provide an interface which lets a Go program map a persistent memory file into its address space. One option could be to support an additional flag MAP_SYNC [2] in the Mmap API to let users map files that specifically support direct-access (DAX). Also, data written to persistent memory can be guaranteed to be made persistent only when they are flushed from the CPU caches as well. An API needs to be made available that can flush a range of memory address using appropriate CPU instructions.
Adding these two support will make it possible for Go programs to easily program persistent memory. It then becomes possible for external packages to be written which provides other features such as memory management, transactions, etc. Note that this is similar to what was implemented in OpenJDK through JEP 352 [1]
There are advantages to providing these features in the core runtime than as an external package - these are the bare-minimal changes required to support persistent memory programming and it becomes much easier later to expand the language to add additional features such as a garbage-collected persistent memory heap, support for transactionally updating persistent memory datastructures, etc. If there is sufficient interest to expand on the initial support added to Go, more features can be added later on.
[1] https://bugs.openjdk.java.net/browse/JDK-8207851
[2] https://man7.org/linux/man-pages/man2/mmap.2.html

Maybe silly question: can't this functionality be implemented in an external package first, such that other external packages can use it and have the capability without requiring language changes?  You refer to the mmap api, that can be used without changes to the (frozen) core syscall package, right?

Maybe silly question: can't this functionality be implemented in an external package first, such that other external packages can use it and have the capability without requiring language changes?  You refer to the mmap api, that can be used without changes to the (frozen) core syscall package, right?

Maybe silly question: can't this functionality be implemented in an external package first, such that other external packages can use it and have the capability without requiring language changes? You refer to the mmap api, that can be used without changes to the (frozen) core syscall package, right?

I was actually suggesting adding capabilities to mmap persistent memory files and adding an API to ensure data persistence be added to core Go. I did not know that the syscall package is frozen.
When we first started working on this project, we explored approaches without requiring any language changes. One of the goals of our project was to expose a programming model that was simple and similar to how Go programs are written currently. But we quickly realized this is not possible if persistent memory programming support is added through an external package
(1) Go is a language with automatic memory management. If persistent memory is managed using a package, then you will need to have explicit APIs for memory allocation and deallocation. Persistent memory is non-volatile. It is easy to miss a free() call and leak memory forever. Also, Go's garbage collector (GC) will not have any visibility into the persistent memory region managed by this library. Any volatile objects pointed from the persistent memory region may be considered as unreachable by the GC.
(2) Supporting transactional data updates is very important for persistent memory programming. We provide a simple txn() { ... } interface for this, where any persistent data updates within the txn(){ ... } block is automatically made crash-consistent. Without compiler changes, enabling such crash-consistent updates is still possible, but the code becomes very verbose. Each data update will now need to be preceded by transactional statements.
So, yes, it is possible to add this support through an external package, but it will be error-prone and expose a difficult programming model. This is why we are proposing native support be added in Go to program persistent memory.

Maybe silly question: can't this functionality be implemented in an external package first, such that other external packages can use it and have the capability without requiring language changes? You refer to the mmap api, that can be used without changes to the (frozen) core syscall package, right?

I was actually suggesting adding capabilities to mmap persistent memory files and adding an API to ensure data persistence be added to core Go. I did not know that the syscall package is frozen.
When we first started working on this project, we explored approaches without requiring any language changes. One of the goals of our project was to expose a programming model that was simple and similar to how Go programs are written currently. But we quickly realized this is not possible if persistent memory programming support is added through an external package
(1) Go is a language with automatic memory management. If persistent memory is managed using a package, then you will need to have explicit APIs for memory allocation and deallocation. Persistent memory is non-volatile. It is easy to miss a free() call and leak memory forever. Also, Go's garbage collector (GC) will not have any visibility into the persistent memory region managed by this library. Any volatile objects pointed from the persistent memory region may be considered as unreachable by the GC.
(2) Supporting transactional data updates is very important for persistent memory programming. We provide a simple txn() { ... } interface for this, where any persistent data updates within the txn(){ ... } block is automatically made crash-consistent. Without compiler changes, enabling such crash-consistent updates is still possible, but the code becomes very verbose. Each data update will now need to be preceded by transactional statements.
So, yes, it is possible to add this support through an external package, but it will be error-prone and expose a difficult programming model. This is why we are proposing native support be added in Go to program persistent memory.

Looking at the low level API for persistent memory, this is, at least on Linux, simply mapping an file. https://pmem.io/pmdk/libpmem/ I don't see why this needs to be harder than https://golang.org/pkg/os/#Open, which requires a Close to be called in the end. You can provide transactions though a mutex-locked callback or a channel of update functions.
I think this is the actual challenge you should take: provide the functionality as an easy to use go module, then  we can consider what pars are better as part of the standard libraries or of the language proper.

Persistent memory technology to my knowledge is encumbered by a net of Intel patents, which in parts extend to the software. I do not think that Go ecosystem would benefit from an  entanglement of active and enforceable software patents of Intel (and more litigous Oracle, too).
@mohit10verma As an excercise: get your patent attorney to help and please conceive an workaround that lets generated code keep a variable of interface type in the "persistent memory"   in a manner that does not infringe on the "pwn it all" US9940229B2.

Looking at the low level API for persistent memory, this is, at least on Linux, simply mapping an file. https://pmem.io/pmdk/libpmem/ I don't see why this needs to be harder than https://golang.org/pkg/os/#Open, which requires a Close to be called in the end. You can provide transactions though a mutex-locked callback or a channel of update functions.

If you look at the PMDK man page, the bulk of the persistent memory initialization is done by the function pmem_map_file(). libpmem does a bunch of things under the covers - identify the mode in which the persistent memory namespace is configured (fsdax or devdax), identify the most optimized cache flush instruction supported on the platform (clwb, clflush, etc.), determine whether memory barriers are necessary to ensure persistence, mmap the persistent memory file with the newly added MAP_SYNC flag so that applications can safely flush the file from userspace, etc. So adding low-level persistent memory support does require a lot more support than just calling os.Open() and os.Close().

I think this is the actual challenge you should take: provide the functionality as an easy to use go module, then we can consider what pars are better as part of the standard libraries or of the language proper.

We did explore adding this functionality as a module but dropped that effort due to reasons I mentioned in an earlier comment.

Sorry, I think I wasn't very clear about what I meant.  I saw that comment, but there you state that the API you would need to provide when doing it as a package wouldn't very nice because you would like to implement a memory like API. So I am suggesting  instead of such a memory like API, a file like API would be just as easy to use for the end user, and possible to implement as a package. It also seems more logical to me. Since persistent memory is persistent, like hard disks or flash drives are, to my mind, it seems more like a storage location for files than something I'd like to use similar to RAM.

Based on my comment last week and the response, it sounds like we should probably decline this full proposal but could easily add the MAP_SYNC flag and perhaps even the "flush memory region" operation.
Since syscall already has Mmap, it would be fine to add MAP_SYNC to syscall on systems that define that flag, even though syscall is frozen to new API. (This is adding a newly defined bit for use with old API.)
And then if you would like to file a proposal for the API for flushing a memory region, that would be fine too.
(Perhaps syscall.Mflush of a []byte like Mprotect/Mlock/etc? Anyway, it should be a separate proposal.)
Thanks.

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

So I am suggesting instead of such a memory like API, a file like API would be just as easy to use for the end user, and possible to implement as a package.

Persistent memory gives a byte-level granularity of read/write access to data stored in it. So in order to get best performance out of it, applications should use direct load/store cpu instructions to read/write persistent memory data. This is achieved by mounting a persistent memory device in DAX mode that avoids the page cache. Applications can then map a file in persistent memory into its address space and directly use it, just like memory. See [1] for some additional FAQs about persistent memory programming. Due to these reasons, a file-like API will be unsuitable for persistent memory programming.
[1] https://software.intel.com/content/www/us/en/develop/articles/persistent-memory-faq.html

Based on my comment last week and the response, it sounds like we should probably decline this full proposal but could easily add the MAP_SYNC flag and perhaps even the "flush memory region" operation.

Thanks @rsc for the updates. Should separate proposals be opened to (1) add a MAP_SYNC flag and (2) add an API that supports flushing a memory address range?

@jerrinsg, yes please add separate proposals for MAP_SYNC and a flushing memory API. Feel free to link to my comment above for context. https://github.com/golang/go/issues/43810#issuecomment-772728874

No change in consensus, so declined.
— rsc for the proposal review group


[HW] San Myint

Name: San Myint
Email<EMAIL_ADDRESS>Linkedin Profile:
Attach the homework screenshots below for both step II and step III:

The CRUD homework is completed here -> https://github.com/DataStax-Academy/Intro-to-Cassandra-for-Developers/issues/39

https://api.badgr.io/public/assertions/LZCuLpikRJuW57Fknj5rQQ?identity__email=san_myint%40trepp.com


Add twitter ingestion engine

switching to this api as it has direct support for wrapping mentions endpoint: https://github.com/PLhery/node-twitter-api-v2/blob/1e4163eda210573dd49b0dcdcb68cd56a374a2b4/doc/v2.md
vs using the query endpoint. the query endpoint has higher rate limits but the mentions endpoint has higher results return amount.
so if we poll every N minutes, we can get more data from mentions than from query


[Removal]:

Hotspot Name
Polished Tin Reindeer
Hotspot b58 Address
112FG8noo6oNoiTTqF1n7gW4ADLFF83g1aTyow45YZe7JECCMwHN
Discord Handle
alanWoz#7541
Hotspot Manufacturer
panther
Removal Reason
I thought the network was blocked by GFW, so I used a VPN. It has now been corrected
Modifications
Yes
Extra forwarders
No
Extra antennas
No
Additional Information
I thought the network was blocked by GFW, so I used a VPN. It has now been corrected

We have taken into account this removal request in the next iteration of the denylist. It may take some time before it's processed. This is an automated comment due to the volume of addition and removal requests. A future tool is in the works to provide more insight into the analysis approach made by the Helium team and community members that maintain this list.


Issues #2, #7, #12, #16, #23 Combined Unit and UI Test Coverage using JaCoCo

#2, #7, #12, #16, #23 :
We can now generate a combined Unit and UI test coverage using JaCoCo.
How to view the UI and Unit test in one JaCoCo report?
Step 1) Open Android Studio
Step 2) Click on Terminal (it's located at the bottom left, last option)
Step 3) Enter the command to execute the report (This takes about 1m 30s, the more test we have, the longer the time gets)
Command to execute combined UI and Unit test in one  JaCoCo report:
gradlew clean combinedTestReportDebug
Important information: This command relies on the fact that all UI and Unit tests are passing. If one fails, it stops and will give an error on the command line as to which test failed its execution!
What does this command do?

It first runs the UI test (you will see your phone magically opening the app and doing things)

It then runs the Unit Test

It finally combines the 2 previous test in one JaCoCo report.

The report is stored under (Project View):
SOEN390-CampusGuideMap\app\build\reports\jacocoCombined\debug\index.html

Commits look good. Amazing work :)
It's just the tests that are failing. Specifically, when I ran the command, at first, I had 3 failures. When I changed the "OK" to "OKAY" (as pointed out in the inline comment) the failures were reduced to 1.
This is the only failure I have after fixing the test:

Error message looks ambiguous. I'd let @makchamp check it out.
*It might be related to testing the title of the alert dialog. "Select Your Calendars". If you remove the fix, you will see the line associated with checking that specific textview fail as well

Commits look good. Amazing work :)
It's just the tests that are failing. Specifically, when I ran the command, at first, I had 3 failures. When I changed the "OK" to "OKAY" (as pointed out in the inline comment) the failures were reduced to 1.
This is the only failure I have after fixing the test:

Error message looks ambiguous. I'd let @makchamp check it out.
*It might be related to testing the title of the alert dialog. "Select Your Calendars". If you remove the fix, you will see the line associated with checking that specific textview fail as well

Actually this issue was related to calendar not being shown on view after selecting the calendar. This has been fixed 😄 awaiting after your review @JeremiahTee 👍


New authentication mechanism

Aloha,
This PR aims to implement a new authentication process (see #10):

Don't rely on the cookie anymore, since it was storing the hash of the password of the user.
Use bcrypt as default hashing function, instead of md5 (now considered weak)…
…but t's still fully backward-compatible with the old format. If one administrator connects to the interface successfully and md5 is still used, it'll automatically be "converted" to one bcrypt hash.
Due to the usage of password_hash, the required PHP version was bumped to PHP 5.6 (it's the current current version in the Debian stable repositories). It may be possible to support older versions using password_compat (I'll add it in another PR it you really want it).

I feel like PHP 5.5 is more common, but Wordpress requires 5.6 as well (but works with 5.2), so it seems reasonable to make 5.6 the minimum requirement. I'll check the PR and merge it later.

I'll take a closer look at the < 5.6 compatibility :-)

I will rebase and work again and finish PR in the next days:

[ ] Take a look at #10 to ensure we are not missing any idea and then close it in favour of this one;
[ ] Provide a polyfill for hash_equals (which is > PHP 5.6) but do not reinvent the wheel with password_hash. The requirement for the PHP engine will only become PHP 5.5;


MFA input validation

Addressing some possible confusion around current “index out of range” runtime errors when selecting an MFA option.

Thanks @Incisive! I updated this and shipped it in #139. Sorry for the delay.


Document Usage

ember-cli-dynamic-forms goes above and beyond the base functionality of alipacajs
We need to set up a site to document usage within an ember application
Info needed for this issue:

list of things to document
documentation host
design for how organize documentation

created an initial site at http://toddjordan.github.io/ember-cli-dynamic-forms
documented the functionality so far.  From now on, new demo/documentation should accompany new features.


Add note that Macro Recorder doesn't exist anymore

Docs Build status updates of commit 21ac12a:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

api/PowerPoint.Adjustments.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 3b6b206:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

api/PowerPoint.Adjustments.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Revamp command line scripts

Replace all command line scripts with two:

[ ] Single state workflow

Run entire workflow for a single state (model_dir)
Flags to skip steps

[ ] All state workflow

Run single state workflow

[ ] Also remove dependency on renv for running command line scripts

I think this is mostly completed by #78


fix: Typos

I set up the collection on Optimism, just waiting for it to be added to quixotic.io so you can actually view it :-). I'll send your NFT as soon as that's ready.

Just minted the first ones! Went for a zombie theme :shrug: Here's yours :-) https://quixotic.io/asset/0x07810385aa29aF1fE6c89f14b2D810954aC78dF8/0

Thanks! :D


ListBox multiple selection (mobile friendly)

Hi, its me again :)
Only a question: Is it possible to make multiple selection works without the needing of pressing CTRL?
I'm developing for a mobile app and it would be nice.
The approach i have thought is making the list permorms allways as if ctrl was pressed if a editor checkbox is selected (i.e. mobile friendly multiple selection)
Thanks for your time.

I've not done too much mobile programming  (actually, I haven't really done any).
What sort of state would you use to detect a regular tap versus a "Control-Tap" to differentiate between a "I only want to select one item" from "I want to select this item, and that item, and that other item"?
I suppose I could add a flag to the ListBox control that removes the "Control key" feature, and automatically detects all selections as a toggle (as if the control key were being pressed).  Would that option work?

Done.   Added a switch to allow the List Items to be toggled Selected/Unselected, as if the Control key were always being pressed.  Update published moments ago.


Update PWF library json and rename pwfconfig.py

Updated to latest version of PWF vendor libraries (playingwithfusion2021.json) which now include Linux and OSX simulation libraries
Renamed pwfconfig.py to venomconfig.py.   This bug was introduced when I changed the configuration in the frc-characterisation GUI from 'PWF' to 'Venom'
Verified libPlayingWithFusion and frc-characterisation changes on robot and in Win32 simulation.

Hi @calcmogul, There's a competition in Quebec starting on Thursday and I'd like them to have this change.   If you're not planning a release this week, is there an easy way to distribute python libraries locally?   There are only 10 teams involved, so it's not the end of the world to touch each machine manually.
I'm not a python guru.  I found where PIP saves the packages locally and can share my library build as a zip file, along with instructions to integrate them.    Is their an easier/better way?

Pip can install from any Git repo: https://stackoverflow.com/a/20101940
You can have them install from the main branch of this repo.

Sounds great, thanks

@CodingWithFusion
I have added some C++ Code to the new tool.
Let me know if you have any suggestions for this implementation.

Thanks!   Looks good so far


The "transformBundle" hook is deprecated. The "renderChunk" hook shou…

The "transformBundle" hook used by plugin es3ify is deprecated. The "renderChunk" hook should be used instead.

Thanks for the commit!

Thanks for the commit!

@sunbing81 can you publish npm?

@sunbing81 can you publish npm?

@sunbing81 can you publish npm?

Done.

@sunbing81 can you publish npm?

Done.

@sunbing81 dist没有到npm啊

@sunbing81 dist没有到npm啊

呃，没看发布内容，现在应该有了


Improved efficiency and code reuse with templates and pure functions

Hello, thanks again for this new series of pull requests. Have you seen that they have all failed CI?

Hello, thanks again for this new series of pull requests.

Not quite new, just splitted the old #182 into multiple PRs.

no template named 'enable_if' in namespace 'std'
error: ‘enable_if’ in namespace ‘std’ does not name a template type

Old versions of C++ should be dropped.

@KOLANICH I find your PR very valuable but I think it may be easier to digest if split into again smaller parts.
@SRombauts What is your take on the points raised by KOLANICH? Lets take one aspect out, templates. Would you agree that adding stronger use of templates is a good thing? I recently had to fix something about the types in Statement::bind(), and I found what KOLANICH also found, that these methods are quite lengthy code for a thing that would become much shorter and cleaner with templates.
Would you accept a PR that adds stronger use of templates (for better code reuse and shortening of the existing code)?


Don't check for updates in CI or bundle exec builds

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I have read the Contribution Guidelines

Issue Description
You have a face paced release schedule... that's great!
For sanity, we use a version locked install of fastlane (via bundler) for both development and CI. In both these cases having fastlane check and report that version upgrades are available is not only useless (and a tiny bit annoying if we cannot update at the moment) but just adds needless to text to our CI build logs.
To put it in perspective we are now basing most of our development on fastlane. For example, starting and stopping local development server environments are now "lanes". So seeing this message for ever command does begin to interfere with your daily tasks.  You've had 3 releases in 3 days... at this point we've almost never run without an upgrade notice.
Detecting that fastlane is in one of these states and forgoing the check for updates, and therefore the display of the notice, would be nice.  A better idea for interactive use may be only do the check once in 24 hours; we get the notice but we're not inundated.
Complete output when running fastlane, including the stack trace and command used
A simple command to stop a local development server environment... ended with a message we cannot act upon (yet).

+-------------------------------------------------------+---------+---------------------------------------+
|                                              Used plugins                                               |
+-------------------------------------------------------+---------+---------------------------------------+
| Plugin                                                | Version | Action                                |
+-------------------------------------------------------+---------+---------------------------------------+
| fastlane-plugin-update_provisioning_profile_specifier | 1.2.0   | update_provisioning_profile_specifier |
| fastlane-plugin-update_project_codesigning            | 0.1.7   | update_project_codesigning            |
+-------------------------------------------------------+---------+---------------------------------------+

[12:11:28]: Driving the lane 'stop_environment' 🚀
[12:11:28]: ----------------------------------
[12:11:28]: --- Step: ensure_xcode_version ---
[12:11:28]: ----------------------------------
[12:11:28]: $ xcversion selected
[12:11:28]: ▸ Xcode 8.2
[12:11:28]: ▸ Build version 8C38
[12:11:28]: Selected Xcode version is correct: 8.2
[12:11:28]: -------------------
[12:11:28]: --- Step: is_ci ---
[12:11:28]: -------------------
[12:11:28]: -------------------
[12:11:28]: --- Step: is_ci ---
[12:11:28]: -------------------
[12:11:28]: ----------------------------
[12:11:28]: --- Step: docker_compose ---
[12:11:28]: ----------------------------
[12:11:28]: $ docker-compose  -f fastlane/edge-dev-compose.yml --project-name dev down
[12:11:29]: ▸ Stopping dev_edge_1 ...
[12:11:29]: ▸ Stopping dev_postgres_1 ...
[12:11:29]: ▸ Stopping dev_apns_1 ...
[12:11:50]: ▸ Removing dev_edge_1 ...
[12:11:50]: ▸ Removing dev_postgres_1 ...
[12:11:50]: ▸ Removing dev_apns_1 ...
[12:11:50]: ▸ Removing network dev_default

+------+----------------------+-------------+
|             fastlane summary              |
+------+----------------------+-------------+
| Step | Action               | Time (in s) |
+------+----------------------+-------------+
| 1    | ensure_xcode_version | 0           |
| 2    | is_ci                | 0           |
| 3    | is_ci                | 0           |
| 4    | docker_compose       | 21          |
+------+----------------------+-------------+

[12:11:50]: fastlane.tools finished successfully 🎉

#######################################################################
# fastlane 2.5.0 is available.
# It is recommended to use the latest version.
# Please update using `bundle update fastlane`.
#######################################################################

2.5.0 2-factor improvements
* Improve support for 2-factor and 2-step verification a lot (#7738)
* Add support for application specific passwords using environment variables (#7738)
* Improved wording and documentation for 2-factor and 2-step (#7738)
* Update checker ensures valid RubyGems source for fastlane updates (#7734)
* Print fastlane path when running `fastlane -v` (#7733)
* Fix crash in _deliver_ when using the new `live` option (#7726)
* Fix USB device discovery to account for USB hubs (#7727)
* Fix USB data parsing problem for DeviceManager (#7725)
* Skip installing plugins when running `fastlane -v` (#7722)
* Add support for uploading proguard mapping files (#7721)

Please update using `bundle update fastlane`

fastlane gems

Gem
Version
Update-Status

fastlane
2.4.0
🚫 Update availaible

Loaded fastlane plugins:

Plugin
Version
Update-Status

fastlane-plugin-update_provisioning_profile_specifier
1.2.0
✅ Up-To-Date

fastlane-plugin-update_project_codesigning
0.1.7
✅ Up-To-Date

Loaded gems

Gem
Version

did_you_mean
1.0.0

bundler
1.13.7

io-console
0.4.6

CFPropertyList
2.3.4

i18n
0.7.0

json
1.8.3

minitest
5.10.1

thread_safe
0.3.5

tzinfo
1.2.2

activesupport
<IP_ADDRESS>

public_suffix
2.0.5

addressable
2.5.0

babosa
1.0.2

claide
1.0.1

fuzzy_match
2.0.4

nap
1.1.0

cocoapods-core
1.1.1

cocoapods-deintegrate
1.0.1

cocoapods-downloader
1.1.3

cocoapods-plugins
1.0.0

cocoapods-search
1.0.0

cocoapods-stats
1.0.0

netrc
0.7.8

cocoapods-trunk
1.1.2

cocoapods-try
1.1.0

colored
1.2

escape
0.0.4

fourflusher
2.0.1

gh_inspector
1.0.2

molinillo
0.5.4

nanaimo
0.2.3

xcodeproj
1.4.2

cocoapods
1.1.1

highline
1.7.8

commander
4.4.3

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20161129

dotenv
2.1.1

excon
0.54.0

multipart-post
2.0.0

faraday
0.10.1

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

faraday_middleware
0.10.1

fastimage
2.0.1

jwt
1.5.6

little-plugger
1.1.4

multi_json
1.12.1

logging
2.1.0

memoist
0.15.0

os
0.9.6

signet
0.7.3

googleauth
0.5.1

httpclient
2.8.3

hurley
0.2

mime-types-data
3.2016.0521

mime-types
3.1

uber
0.0.15

representable
2.3.0

retriable
2.1.0

google-api-client
0.9.20

mini_magick
4.5.1

multi_xml
0.6.0

plist
3.2.0

rubyzip
1.2.0

security
0.1.3

slack-notifier
1.5.1

terminal-notifier
1.7.1

unicode-display_width
1.1.2

terminal-table
1.7.3

word_wrap
1.0.0

rouge
1.11.1

xcpretty
0.2.4

xcpretty-travis-formatter
0.0.4

fastlane-plugin-update_project_codesigning
0.1.7

fastlane-plugin-update_provisioning_profile_specifier
1.2.0

xcode-install
2.1.0

You can either set FASTLANE_SKIP_UPDATE_CHECK or FASTLANE_HIDE_CHANGELOG (one skip the update checks all together, the other will just hide the changelog.

@KrauseFx Thanks for the info.  I can update our CI easily.  For local, is there a easy place to put this in our fastlane setup or do we resort to .bashrc/.zshrc?

.bashrc/.zshrc works 👍


Error while exporting

Everytime i try to export blender shows me the error "no objects of type: drawable or drawable dictionary or bound composite to export"

Read the Sollumz wiki.


add Highmaps support

I want to use Highmaps with React.
This allows users to render maps.
Usage:
var Highcharts = require('react-highcharts');
require('react-highcharts/highmaps');

<Highcharts type="map" config={config}></Highcharts>

Hi Nash, Thanks a lot for the PR, did you have a chance to test it?

Won't this cause this issue?
https://github.com/kirjs/react-highcharts/issues/9

I guess that's something I'd have to do with more.js to close #9
That would be an API change and would require version bump.
Feel free to do a PR, or I'll try to do that over the weekend

upd: Josh, I misread your comment.
Highcharts is a bit confusing in this way: more and map, are add-ons to highcharts, while highstock is a separate entity.
I just have to get it all together and figure out all possible permutations, something I'm planning to do over the weekend.

Hi @kirjs, thanks for your comment, and I'm sorry for the late reply.
I tested my code in my project by installing react-highcharts with npm i nash716/react-highcharts
I think that my code doesn't cause #9.
a.jsx
var Highcharts = require('react-highcharts');

b.jsx
var Highcharts = require('react-highcharts');
require('react-highcharts/highmaps');

c.jsx
var Highcharts = require('react-highcharts');
require('react-highcharts/highmaps');

react-highcharts/highmaps was required twice. On the other hand, a.jsx requires react-highcharts only. It works without any errors.
But the big overhaul project is in progress (#20), I think I should wait for it to finish.
What do you think?

Hi Nash, no worries, I agree on #20, let's get it in, and see how the merge would look like.

@kirjs can you update the readme.md to show how to use highmap with a map?

The pull request has not been merged yet.
@nash716 Can you get it on top of master and see if it needs some updating?

Hi,
I've been tied up with work, couldn't grasp the change.
And what is worse, I'm busy for a while...
I'll make time to update my code.
@thientran1707
If you are in a hurry, you can use this feature for now.
Please run $ npm i nash716/react-highcharts.
Readme is here.

Hi,
Any chance this pull request will be merged soon?
Thanks,
Gil

I tried to update my code, but some problems were found.
highmaps-release doesn't contain map module.
highmaps-release contains highmaps.src.js, and highmaps.src.js has Highcharts.
The map module is not available.
Codes like below will cause this ( #39 )
a.jsx
var Highcharts = require('react-highcharts');

b.jsx
var Highmaps = require('react-highcharts/highmaps');

Fixed in #45


TypeError: Cannot read property 'id' of undefined

https://sentry.io/tendermint/cosmos-voyager/issues/773149452/?referrer=github_plugin
TypeError: Cannot read property 'id' of undefined
  at Store.addToCart (/private/var/folders/kn/6djg7f2932353fwm44_0cxjc0000gn/T/AppTranslocation/D0DE78EF-BAB8-4ED5-8A69-4DA37485BB4F/d/Cosmos%20Voyager.app/Contents/Resources/app/dist/renderer.js:111984:24)
  at wrappedMutationHandler (/private/var/folders/kn/6djg7f2932353fwm44_0cxjc0000gn/T/AppTranslocation/D0DE78EF-BAB8-4ED5-8A69-4DA37485BB4F/d/Cosmos%20Voyager.app/Contents/Resources/app/dist/renderer.js:5868:13)
  at commitIterator (/private/var/folders/kn/6djg7f2932353fwm44_0cxjc0000gn/T/AppTranslocation/D0DE78EF-BAB8-4ED5-8A69-4DA37485BB4F/d/Cosmos%20Voyager.app/Contents/Resources/app/dist/renderer.js:5561:7)
  at Array.forEach (<anonymous>)
  at None (/private/var/folders/kn/6djg7f2932353fwm44_0cxjc0000gn/T/AppTranslocation/D0DE78EF-BAB8-4ED5-8A69-4DA37485BB4F/d/Cosmos%20Voyager.app/Contents/Resources/app/dist/renderer.js:5560:11)
...
(12 additional frame(s) were not displayed)

addToCart is actual used and removing it produces side effects. needs to be investigated.

Deprioritized because it only happened once.


refactor extended resource with kubernetes.io namespace

Signed-off-by: 佑祎<EMAIL_ADDRESS>Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?
#83

Ⅲ. Describe how to verify it
Ⅳ. Special notes for reviews
V. Checklist

[x] I have written necessary docs and comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in make test

/lgtm
/approve


Restrict TeamMember permission to link, create or clone survey to engagement. Must be assigned to it

TM can't create a survey.


DataGrid Cell edit Theme issue

It seems DataGrid cell edit context menu ignores the theme

@thedimm This is fixed in latest source (also available at NuGet as pre-release). Keep in mind that there are many breaking changes in there.
https://github.com/MahApps/MahApps.Metro/pulls?q=is%3Apr+is%3Aclosed+label%3A"Breaking+Change"+milestone%3A2.0.0
https://github.com/MahApps/MahApps.Metro/issues?q=is%3Aissue+is%3Aclosed+milestone%3A2.0.0+label%3A"Breaking+Change"

Hello plunker76 . Any plans to fix in 1.6 ?
IF not when will 2.0.0 be released?

@thedimm Due date is May 01, 2020 (that's the plan, if there is no other catastrophe)


Sharding for instrumentation tests

So currently you guys support env vars which is great. The issue is if my device list has more then 1 device and i pass the params numShards=2 shardIndex=0 it will only run half my test suite on both devices. Im asking if there is a way to give separate shards to each device.
Example:

Device 1 (Nexus5) has env vars numShards=2 shardIndex=0
Device 2 (Nexus6P) has env vars numShards=2 shardIndex=1

Maybe this can be a parameter under the instrumentation tab like there is for Test targets

Hi @gpowell13,
Nice catch, unfortunately, we do not support test sharding, yet. Please open a feature request on https://discuss.bitrise.io/c/feature-request if there is not one already so that we can contact you once the feature request is triaged.
Let me know if you have any issues with creating the feature request.
Thanks. :)


SQLite over SFTP?

Is it possible to have a SQLite connection over SFTP/SCP/FTP ?
I have a SQLite database stored on a server. Right now I'm transferring the SQLite file to my computer (via SFTP) in order to access it with DBeaver, then modify the data, then I transfer the file back to the server. That would be neat if it was possible to access the file over some protocol like SFTP.

SQLite doesn't have network protocol at all. It is an embedded database so it is not possible to work with it by wire in any fashion.
Probably there are some existing "wrappers" which can be installed on server side to operate with SQLite remotely but I'm not aware about that. And in any case it is beyond of DBeaver functionality.

Thanks for the reply, that makes sense.

SQLite seem to be working with sshfs FUSE file system. Although super slow and inefficient and probably WAL won't be working as it's using memory mapped file. But might work in an emergency case.


Render entire container scroll height for VirtualScroll

Hi Brian,
I'm currently using VirtualScroll where I am manually changing the scrollTop when the user clicks on a button that should scroll them to a particular row. The reason I'm not using the provided scrollToIndex prop is because the scroll that I need to do requires animation.
However this becomes an issue if there are tons of rows being rendered and I need to animate the scrollTop to something not yet visible (ie. the last row). Because VirtualScroll only renders what is visible to the user, The container has a inaccurate scroll height representation; and only when the user begins to scroll further down, does the scroll height begin to increase and increase.
If I already know the expected scroll height of the container, would there be a way to add a prop for me to pass it down, so that the container would be the correct scroll height, but still render only the visible rows, the rest of the scroll area would just be whitespace.
In the Grid component that VirtualScroll uses, there is a totalRowsHeight variable to set the height of the of the div with the class Grid__innerScrollContainer.
const totalRowsHeight = this._rowSizeAndPositionManager.getTotalSize();
Ideally, it would be great if we could so something like:
const totalRowsHeight = this.props.totalRowsHeight || this._rowSizeAndPositionManager.getTotalSize();
PS. I love this library, it's slick as hell!
Best,

Johnny

Hey @johnnyji,
Thanks for the compliment. :)
When you say you already know the total height- do you mean your rows are all the same height? Or they're varying height, but not dynamically measured?

@bvaughn
In my case, all the rows are the same height, and I know exactly how many rows there will be. Therefore I can do the calculation on my end to get the total height of all the rows.
If I can give VirtualScroll the total height of all the rows, and that in turn passes it down to Grid, then on mount, the VirtualScroll should be able to render the correct overflow-y for all the rows, instead of the overflow incrementally increasing as I scroll further down.
I hope I'm making sense with my description haha

I see. Maybe I'm missing something, but if your rows are all the same height- then this line in Grid should set the estimatedRowSize to be equal to your specified rowHeight which should cause the "estimated" height returned by getTotalSize to be the exact, correct height, no?

@bvaughn Whoops. I got mixed up in my own head. My rows are NOT the same height. But I can calculate the sum height of all the rows.

Heh heh, ok. That makes more sense then.
Let me think about this some. I'm a little wary of adding a totalRowsHeight property. It seems a little too easy to mis-use and break things.
Any chance you have a Plunker or something you could point me at so I could get a sense of how "bad" this gradual resizing is? My hope was that using reasonable estimatedRowSizes would make the resizing pretty subtle. How much do your rowHeights vary? Are you providing a custom estimatedRowSize or just using the default? :)

Check out new release 7.2.0.
Ended up going a slightly different way with this but I think it solves your problem while being a little safer. Updated docs here: https://github.com/bvaughn/react-virtualized/blob/master/docs/Grid.md#measureallcells
Basically there's a new method on Grid (measureAllCells) and FlexTable and VirtualScroll (measureAllRows) that allows you to force-measure.

@bvaughn 👏 👏 👏  Far better than I originally thought. Thanks a lot for the quick response and wicked library 👍


Add service management functions

Discussed in https://github.com/orgs/dart-windows/discussions/681

Originally posted by mateojc007 April 11, 2023
Hi,
Congratulations for your lib.
I'm currently working on a flutter app based on wireguard VPN.
In order to make it work on windows I need to access couple of advapi32.dll functions :

OpenSCManagerW
OpenServiceW
CreateServiceW
CloseServiceHandle
ControlService
QueryServiceStatus
DeleteService
StartServiceW
ChangeServiceConfig2

I'm currently using FFI to call this functions but I was wondering if you have any plan to support these functions in the short term.
Best regards

@mateojc00 Included in v5.0.2. Thanks for the report!


Send request map failed on start/reboot

Describe the bug
Not sure if its affecting other people but for me after updating to the latest beta v2.0.0b16 each time I start/reboot hass I'm getting the error below which causes the integration to fail/not load properly. Though the workaround I have found is to manually reload the integration after the error, then it works just fine.
To Reproduce
hass on start/restart will throw the Send request map failed: Unable to discover the device over cloud error
Expected behavior
Integration should start normally without needing to be reloaded manually
Additional Information (please complete the following information)

Model Name [e.g. dreame.vacuum.p2028]: dreame.vacuum.r2228o
Firmware Version [e.g. 1156]: 4.3.9_3204
Home Assistant Version: 2024.7.4
Configuration Type [With or without map support]: with low res map
Errors or warnings shown in the HA logs (if applicable):

2024-08-13 21:55:41.119 WARNING (SyncWorker_1) [custom_components.dreame_vacuum.dreame.map] Send request map failed: Unable to discover the device over cloud
2024-08-13 21:55:41.125 ERROR (Thread-12 (_api_task)) [root] Uncaught thread exception
Traceback (most recent call last):
File "/usr/local/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/config/custom_components/dreame_vacuum/dreame/protocol.py", line 112, in _api_task
item[0](self._api_call(item[1], item[2], item[3]))
File "/config/custom_components/dreame_vacuum/dreame/protocol.py", line 388, in
lambda api_response: callback(
^^^^^^^^^
File "/config/custom_components/dreame_vacuum/dreame/protocol.py", line 1206, in cloud_callback
raise DeviceException("Unable to discover the device over cloud") from None
custom_components.dreame_vacuum.dreame.exceptions.DeviceException: Unable to discover the device over cloud

Ah yes think you're spot on must have been some funky connection issue from on my side or Dreame's SG servers (now to think of it couple of times I could not get the map to load even on Dreame's app). I can no longer reproduce this error now as I did last couple of days. Thanks!


Programmatically update parameter value does not update widget

See the following code.  xystream.event(y=-0.3) updates y value to -0.3, but the slider widget is not updated accordingly.
from holoviews.streams import Stream
import param
import paramnb

class XY(Stream):
    x = param.Number(default=0.0, bounds=(-0.5, 0.5))
    y = param.Number(default=0.0, bounds=(-0.5, 0.5))

    def event(self, **kwargs):
        super(XY, self).event(**kwargs)
        print(self.contents)

xystream = XY()
paramnb.Widgets(xystream, callback=xystream.event, on_init=True)

# does not update y slider
xystream.event(y=-0.3)

Solved by doing
w = paramnb.Widgets.instance()
w(xystream, callback=xystream.event, on_init=True)
w.widget('y').value = xystream.contents['y']

Not sure if this is the right way to do it?


Add AWS Zone support (#35)

Add new variable for AWS Availability Zone, with default a (as implied it seems)
Specify zone explicitly in MachineOptions
Update documentation
Update changelog

This fixes #35

Thanks for the contribution. Trying to have a look in the coming week.

@declension thanks for the contribution. I added the zone paramater to the runner template, it was missing. I have also renamed the parameter in the template.

Cool, thanks!