Investigate using Managed/Shared texture storage mode in Metal for direct uploads

Already got it to work veldrid-side with the following diff:
diff --git a/src/Veldrid/MTL/MTLGraphicsDevice.cs b/src/Veldrid/MTL/MTLGraphicsDevice.cs
index c864c81..7369954 100644
--- a/src/Veldrid/MTL/MTLGraphicsDevice.cs
+++ b/src/Veldrid/MTL/MTLGraphicsDevice.cs
@@ -394,26 +394,21 @@ private protected override void UpdateBufferCore(DeviceBuffer buffer, uint buffe
             uint arrayLayer)
         {
             MTLTexture mtlTex = Util.AssertSubtype<Texture, MTLTexture>(texture);
-            if (mtlTex.StagingBuffer.IsNull)
-            {
-                Texture stagingTex = ResourceFactory.CreateTexture(new TextureDescription(
-                    width, height, depth, 1, 1, texture.Format, TextureUsage.Staging, texture.Type));
-                UpdateTexture(stagingTex, source, sizeInBytes, 0, 0, 0, width, height, depth, 0, 0);
-                CommandList cl = ResourceFactory.CreateCommandList();
-                cl.Begin();
-                cl.CopyTexture(
-                    stagingTex, 0, 0, 0, 0, 0,
-                    texture, x, y, z, mipLevel, arrayLayer,
-                    width, height, depth, 1);
-                cl.End();
-                SubmitCommands(cl);
-
-                cl.Dispose();
-                stagingTex.Dispose();
+
+            mtlTex.GetSubresourceLayout(mipLevel, arrayLayer, out uint dstRowPitch, out uint dstDepthPitch);
+
+            if (!mtlTex.DeviceTexture.IsNull)
+            {
+                var origin = new MTLOrigin(x, y, z);
+                var size = new MTLSize(width, height, depth);
+
+                if (texture.Type != TextureType.Texture2D)
+                    throw new NotSupportedException("This operation is only supported on 2D textures for now.");
+
+                mtlTex.DeviceTexture.replaceRegion(new MTLRegion(origin, size), (UIntPtr)mipLevel, (UIntPtr)arrayLayer, source.ToPointer(), (UIntPtr)(width * 4), UIntPtr.Zero);
             }
             else
             {
-                mtlTex.GetSubresourceLayout(mipLevel, arrayLayer, out uint dstRowPitch, out uint dstDepthPitch);
                 ulong dstOffset = Util.ComputeSubresourceOffset(mtlTex, mipLevel, arrayLayer);
                 uint srcRowPitch = FormatHelpers.GetRowPitch(width, texture.Format);
                 uint srcDepthPitch = FormatHelpers.GetDepthPitch(srcRowPitch, height, texture.Format);
diff --git a/src/Veldrid/MTL/MTLTexture.cs b/src/Veldrid/MTL/MTLTexture.cs
index e01620e..5da47d6 100644
--- a/src/Veldrid/MTL/MTLTexture.cs
+++ b/src/Veldrid/MTL/MTLTexture.cs
@@ -63,7 +63,7 @@ public MTLTexture(ref TextureDescription description, MTLGraphicsDevice _gd)

             if (Usage != TextureUsage.Staging)
             {
-                MTLStorageMode = isDepth && _gd.PreferMemorylessDepthTargets ? MTLStorageMode.Memoryless : MTLStorageMode.Private;
+                MTLStorageMode = isDepth && _gd.PreferMemorylessDepthTargets ? MTLStorageMode.Memoryless : MTLStorageMode.Shared;

                 MTLTextureDescriptor texDescriptor = MTLTextureDescriptor.New();
                 texDescriptor.width = (UIntPtr)Width;

But quick profiling sessions with this test case show pretty interesting results.
upload staging texture + copy command:

Managed/Staging texture & direct write using replaceRegion:

As can be seen above, keeping texture as Private and using copy command performs 2x faster than making it Managed/Shared and updating it directly. This might have a relation with the fact that I'm testing this on an Apple GPU.

@peppy interested to see results of this on non-apple GPUs (see diff above).

keeping texture as Private and using copy command performs 2x faster than making it Managed/Shared and updating it directly

private is documented as performing better than shared buffers due coherency not being required: https://developer.apple.com/documentation/metal/mtlstoragemode/private . It's also the reason I asked you to test it for VBOs back a while ago, but at the time you mentioned there was no discernible change iirc.

I've been reconsidering that since shared/managed storage is the default for MTLTextures, but I think this boils down to how much we upload to the texture from CPU as opposed to drawing with it in the GPU.
I've noticed a couple of flags that have an effect on this, both negatively and positively, so I've went ahead and made screenshots of the frame graphs with each one of these flags (note all of them use managed/shared storage mode).

untracked resource
no gpu optimisation

untracked resource

write-combined cache

no gpu optimisation

According to the above screenshots and furthermore checking, marking managed/shared texture resource as untracked helps with the upload overhead, but still doesn't reach the performance of https://github.com/ppy/osu-framework/pull/5911. Those flags seem pretty interesting to play with elsewhere, but initial attempt don't show anything immediately good from them, so I will leave it at that.
Going to close this since managed/shared textures aren't beneficial for us compared to private.


Implementation without TCP-Modbus (incl. demo code)

Is your feature request related to a problem? Please describe.
Currently you have to enable TCP Modbus, which can only be activated with technical-login
Describe the solution you'd like
It is possible to fetch all data that are needed without TCP Modbus, by using python and a simple ethernet connection.
Describe alternatives you've considered
I debugged the web-interface of IDM: The only thing the python script needs, is the ip-address and the password for login. The password can be set at display panel on the idm heatpump (without technical-login).
The cool thing is, that via web-login, automatically all "heating circuits" / "Heizkreise" will be detected.
Here is a demo, which isn't working on my home-assistent yet, but the python script is working and gets all data which are needed. The trick is to extract the " csrf_token" on website login:
import logging
from datetime import timedelta
import requests

from homeassistant.components.sensor import PLATFORM_SCHEMA
import homeassistant.helpers.config_validation as cv
from homeassistant.const import (
    CONF_pin,CONF_ipaddress, CONF_SCAN_INTERVAL, CONF_RESOURCES)
from homeassistant.util import Throttle
from homeassistant.helpers.entity import Entity

_LOGGER = logging.getLogger(__name__)

MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=300)

SENSOR_PREFIX = 'Heating '

SENSOR_TYPES = {
    'mode': ['mode', '', 'mdi:settings'],
    'circuit_mode': ['circuit mode', '', 'mdi:settings'],
    'errors': ['errors', '#', 'mdi:alert-circle'],
    'heat_quantity': ['heat quantity', 'kWh', 'mdi:radiator'],
    'outside_temp': ['outside temp.', '°C', 'mdi:thermometer'],
    'forerun_temp_actual': ['forerun temp. (actual)', '°C', 'mdi:temperature-celsius'],
    'forerun_temp_target': ['forerun temp. (target)', '°C', 'mdi:temperature-celsius'],
    'room_temp_target': ['room temp. (target)', '°C', 'mdi:thermometer'],
    'return_flow_temp': ['return flow temp.', '°C', 'mdi:temperature-celsius'],
    'hygienic_temp': ['hygienic temp.', '°C', 'mdi:temperature-celsius'],
    'water_temp': ['water temp.', '°C', 'mdi:temperature-celsius'],
}

PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
    vol.Required(CONF_PIN): cv.string,
    vol.Required(CONF_IPADDRESS): cv.string,
    vol.Required(CONF_RESOURCES, default=[]):
    vol.All(cv.ensure_list, [vol.In(SENSOR_TYPES)]),
})

def setup_platform(hass, config, add_entities, discovery_info=None):
    _LOGGER.debug("Setup IDM Terra sensors")

    # ipaddress    = config.get(CONF_IPADDRESS)
    # pin          = config.get(CONF_PIN)
    # scanInterval = config.get(CONF_SCAN_INTERVAL)

    ipaddress = "<IP_ADDRESS>"
    pin       = "666"

    try:
        data = IdmData(ipaddress, pin)
    except requests.exceptions.HTTPError as error:
        _LOGGER.error(error)
        return False

    entities = []

    for resource in config[CONF_RESOURCES]:
        sensor_type = resource.lower()

        if sensor_type not in SENSOR_TYPES:
            SENSOR_TYPES[sensor_type] = [
                sensor_type.title(), '', 'mdi:flash']

        entities.append(IdmHeatingSensor(data, sensor_type))

    add_entities(entities)

class IdmData(object):
    def __init__(self, ipaddress, pin):
        self._ipaddress = ipaddress
        self._pin       = pin

        self.info              = None
        self.heatpump          = None
        self.pv                = None
        self.stat_baenergyhp   = None
        self.stat_bapv         = None
        self.stat_heatpump     = None
        self.stat_amountofheat = None

    @Throttle(MIN_TIME_BETWEEN_UPDATES)
    def update(self):
        _LOGGER.debug("updating IDM data")

        headers = {
            "Connection": "Keep-Alive",
        }
        params = {
            "pin": self.pin
        }

        self.info              = None
        self.heatpump          = None
        self.pv                = None
        self.stat_baenergyhp   = None
        self.stat_bapv         = None
        self.stat_heatpump     = None
        self.stat_amountofheat = None

        try:
            with requests.Session() as s:
                # get cookie
                r = s.post("http://" + self._ipaddress, headers=headers)

                # enter pin
                r = s.post("http://" + self._ipaddress +"/index.php", headers=headers, data=params)

                #extract csrf_token
                keyname = "csrf_token="
                txt     = r.text
                start = txt.find(keyname) + len(keyname) + 1 # get start of csrf_token entry

                j=start;

                csrf_token =""
                while(j < len(txt)):
                    csrf_token = csrf_token + txt[j]
                    j = j + 1

                    if(txt[j] == '"'):
                        break

                headers["CSRF-Token"] = csrf_token

                #read data
                self.info              = s.get("http://" + self._ipaddress +"/data/info.php", headers=headers).json()
                self.heatpump          = s.get("http://" + self._ipaddress +"/data/heatpump.php", headers=headers).json()
                self.stat_baenergyhp   = s.get("http://" + self._ipaddress +"/data/statistics.php?type=baenergyhp", headers=headers).json()
                self.stat_bapv         = s.get("http://" + self._ipaddress +"/data/statistics.php?type=bapv", headers=headers).json()
                self.stat_heatpump     = s.get("http://" + self._ipaddress +"/data/statistics.php?type=heatpump", headers=headers).json()
                self.stat_amountofheat = s.get("http://" + self._ipaddress +"/data/statistics.php?type=amountofheat", headers=headers).json()

                return True
        except requests.exceptions.RequestException as exc:
            _LOGGER.error("Error occurred while fetching data: %r", exc)
            self.data = None
            return False

class IdmHeatingSensor(Entity):

    def __init__(self, data, sensor_type):
        self.data   = data
        self.type   = sensor_type
        self._name  = SENSOR_PREFIX + SENSOR_TYPES[self.type][0]
        self._unit  = SENSOR_TYPES[self.type][1]
        self._icon  = SENSOR_TYPES[self.type][2]
        self._state = None

    @property
    def name(self):
        return self._name

    @property
    def icon(self):
        return self._icon

    @property
    def state(self):
        return self._state

    @property
    def unit_of_measurement(self):
        return self._unit

    def update(self):
        self.data.update()
        heatingData = self.data.data

        try:

# self.info              = None
# self.heatpump          = None
# self.pv                = None
# self.stat_baenergyhp   = None
# self.stat_bapv         = None
# self.stat_heatpump     = None
# self.stat_amountofheat = None

            if self.type == 'mode':
                self._state = heatpump["system"]["srcmode"]

            elif self.type == 'circuit_mode':
                for key, value in circuitModeDictionary.items():
                    heatingData["circuits"][0]["mode"] = heatingData["circuits"][0]["mode"].replace(key, value)
                self._state = heatingData["circuits"][0]["mode"]

            elif self.type == 'errors':
                self._state = int(heatingData["error"])

            elif self.type == 'heat_quantity':
                self._state = float(heatingData["sum_heat"].rstrip(" kWh"))

            elif self.type == 'outside_temp':
                self._state = float(info["out"])

            elif self.type == 'forerun_temp_actual':
                self._state = float(heatingData["circuits"][0]["temp_forerun_actual"].rstrip(" °C"))

            elif self.type == 'forerun_temp_target':
                self._state = float(heatingData["circuits"][0]["temp_forerun"].rstrip(" °C"))

            elif self.type == 'room_temp_target':
                self._state = float(heatingData["circuits"][0]["temp_room_value"])

            elif self.type == 'return_flow_temp':
                self._state = float(heatingData["temp_heat"].rstrip(" °C"))

            elif self.type == 'hygienic_temp':
                self._state = float(heatingData["temp_hygienic"].rstrip(" °C"))

            elif self.type == 'water_temp':
                self._state = float(heatingData["temp_water"].rstrip(" °C"))
        except ValueError:
            self._state = None

A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

I know it would technically be possible to scrape the data from the web interface via HTTP requests. However, there are multiple reasons why I didn't implement it this way.
The main reason is that the HTTP API you're using is not a public API AFAIK and could change in arbitrary and incompatible ways with any update IDM deploys. My understanding is that this API is designed purely for the web interface, so IMO it's not unlikely that a small change in the web interface could also cause a change in the API.
The Modbus API on the other hand is explicitly meant for communication with external systems and will remain compatible in future versions. AFAIK Modbus is also used by some major 3rd party integrations like the integration of Fronius solar inverters, so it is unlikely IDM would introduce a breaking change.
Another reason is that I didn't want to put in the work of reverse engineering the HTTP API, since the Modbus API is well documented by IDM.
In short, using the HTTP API is possible yes, but out of scope for this integration. If you do not want to enable Modbus for whatever reason, you'll have to write your own separate integration. If you want to do that, you may want to take a look at the Home Assistant Developer Docs and the HACS Docs. In particular you should probably look into the DataUpdateCoordinator, since the heat pump provides quite a lot of sensors which can be fetch together. You may also wanna look at httpx for an HTTP library with async support.


encoding/json json.Unmarshel cannot deal with leading zeros

In the process of dealing with the readings coming from a electricity meter I was trying to process these json strings:
{"datetime":"2022-02-14T19:53:08Z","import":0000504.720,"export":0117950.472}
unmarshel was returning a empty struct
Changing the input data to:
{"datetime":"2022-02-14T19:53:08Z","import":504.720,"export":117950.472}
Resolved the problem
package main
import (
"fmt"
"encoding/json"
"time"
)
func main() {
type HydroMeter struct {
	M1datetime	string `json:"datetime"`
	M1import	float32	`json:"import"`
	M1export	float32 `json:"export"`
}
hydrometer1 := `{"datetime":"2022-02-14T19:53:08Z","import":0000504.720,"export":0117950.472}`
var hydroMeter HydroMeter
json.Unmarshal([]byte(hydrometer1), &hydroMeter)
fmt.Println(hydroMeter)

hydrometer2 := `{"datetime":"2022-02-14T19:53:08Z","import":504.720,"export":117950.472}`
json.Unmarshal([]byte(hydrometer2), &hydroMeter)
fmt.Println(hydroMeter)
fmt.Println(time.Parse(time.RFC3339, hydroMeter.M1datetime))

}
$ go run unmarshal.go
{ 0 0}
{2022-02-14T19:53:08Z 504.72 117950.47}
2022-02-14 19:53:08 +0000 UTC

You aren't checking the errors: https://go.dev/play/p/-_MM7KETebL
JSON does not allow leading zeros in natural numbers, like 001. Leading zeros are only allowed in cases where you need them, like 0.5. See https://www.json.org/json-en.html and https://stackoverflow.com/questions/27361565/why-is-json-invalid-if-an-integer-begins-with-a-leading-zero.
Int he future, see https://golang.org/wiki/Questions.

thanks Daniel @mvdan


N04 Sub05 adds extra whitespace to end of line inside inner block

I noticed that the new option for adding padding on the right also introduces a significant amount of padding inside the inner-most block.
Code:
# hello
for i in range(10):
    for j in range(20):
        print("Hello world, from ({},{})".format(i,j))
        # expand block
        count += 1

N04 Sub05 -> 0:

N04 Sub05 -> 4:

I think it might be interacting poorly with N24:

That's because the calculation formula for N04 Sub05 is like this:
rightPaddingIncrement = (maxDepth - currDepth) * minDistanceBetweenRightEdges
It is a fast formula, performance is more optimized with it. If we want more precise formula, then I guess we need to make much more complex and CPU heavy calculations. I think current formula is good enough. I hope it's ok for you.
You mentioned N24, and yeah, it will make some effect on the formula because if you enable round brackets and maybe also square brackets, it means you are creating more blocks. More blocks often means some changes in depth indexes, so it will effect the formula.


Submenu not focused when using keyboard navigation

I am using a component for the Submenu label

                   <SubMenu
                        openTrigger={triggerByClick ? 'clickOnly' : undefined}
                        key={itemId}
                        className="navigation__main-menu-link menu-link"
                        label={(
                             <>

                                     {displayName}

                                   {supportingText &&
{supportingText}
}

                             </>
                        )}
                        gap={5}
                        onMenuChange={(e) => e.open && onItemSelect(itemId)}
                    >
                        //.....
                    </SubMenu>

But when I use the keyboard to navigate to a submenu item, it is not focusing on it. Any ideas how to fix this?

With the above JSX provided to a submenu label, focus will be placed on the menu item div which wraps the label.

Focus will be on the highlighted element that has class="szh-menu__item szh-menu__item--submenu" in the screenshot.


MapPath problem with JPA / Hibernate  in v4.1.3

I have a strange problem with a MapPath mapping. I don't know if it's  something on my side, so I'm posting it here (sorry for using quotes instead of code, it was not formatting as it should for some reason!)
I have this kind of mapping:

@ElementCollection
@MapKeyJoinColumn(name = "workMode")
@CollectionTable(name = "WorkByMode", joinColumns = { @JoinColumn(name = "session_id") })
@Column(name = "workCount")
protected Map<WorkMode, Integer> workByMode

When I try to filter a query with the map keys, allowing to have one of multiple values using a boolean builder:

BooleanBuilder where = new BooleanBuilder();
for (WorkMode mode : modes) {
where.or(myEntity.workByMode.containsKey(mode));
}

The SQL that ends up being generated has a left outer join with WorkByMode (which is great) but then an inner join with WorkMode

left outer join
WorkShiftStatistics_workByMode workbymode2_
on workshifts0_.id=workbymode2_.session_id
and (
workbymode2_.workMode=?
)
inner join
WorkMode workmode3_
on workbymode2_.workMode=workmode3_.code
left outer join
WorkShiftStatistics_workByMode workbymode4_
on workshifts0_.id=workbymode4_.session_id
and (
workbymode4_.workMode=?
)
inner join
WorkMode workmode5_
on workbymode4_.workMode=workmode5_.code
where
(
workbymode2_.count is not null
or workbymode4_.count is not null
)

So if I only have one of the given keys, the second inner join makes the result empty... if it were a left join it wouldn't.
I don't know if other sql engines use another precedence, but I'm using sqlserver and this is not working.
Is it me? Can I avoid the second inner join somehow?

Generated SQL is a tough issue in Querydsl JPA, but we'll see if it's something on our end, on Hibernate's end or your query.
Could you maybe post our generated JPQL?
Did you already make aliases when joining?
Maybe you need to be explicit with what path you want.
For containsKey we join to the property's parent
BooleanBuilder where = new BooleanBuilder();
for (WorkMode mode : modes) {
    where.or(myEntity.workByMode.containsKey(mode));
    //      ^  ^  ^  ^  ^  ^  ^
    // maybe use an alias here?
}

Maybe we need to see more of the query, is that possible?
An example project that has the issue is also good (would also help if you need to take the issue somewhere else).

Here's more detail
the WorkShiftStatistics Entity
@Entity
public class WorkShiftStatistics {
    @Id
    protected long id;

    private boolean active;

    @ElementCollection
    @MapKeyJoinColumn(name = "workMode")
    @CollectionTable(name = "WorkShiftStatistics_workByMode", joinColumns = { @JoinColumn(name = "statistics_id") })
    @Column(name = "count")
    protected Map<WorkMode, Integer> workByMode = Maps.newHashMap();

    ...
}

the query builder
@Override
public HibernateQuery createQuery(List<WorkMode> modes) {
    BooleanBuilder whereBuilder = new BooleanBuilder();
    for (WorkMode mode : modes) {
        whereBuilder.or(workShiftStatistics.workModes.contains(mode.getCode()));
    }
    return newQuery()
            .from(workShiftStatistics)
            .leftJoin(workShiftStatistics.user, user)
            .where(whereBuilder);
}

the output sql

Hibernate:
    /* select
        count(workShiftStatistics)
    from
        WorkShiftStatistics workShiftStatistics
    left join
        workShiftStatistics.user as user
    left join
        workShiftStatistics.workByMode as workShiftStatistics_workByMode_1992797668 with key(workShiftStatistics_workByMode_1992797668) = ?1
    left join
        workShiftStatistics.workByMode as workShiftStatistics_workByMode_25899661 with key(workShiftStatistics_workByMode_25899661) = ?2
    where
        (
            workShiftStatistics_workByMode_1992797668 is not null
            or workShiftStatistics_workByMode_25899661 is not null
        )
    */

    select
            count(workshifts0_.id) as col_0_0_
        from
            WorkShiftStatistics workshifts0_
        left outer join
            EoxUser user1_
                on workshifts0_.user_id=user1_.id
        left outer join
            WorkShiftStatistics_workByMode workbymode2_
                on workshifts0_.id=workbymode2_.statistics_id
                and (
                    workbymode2_.workMode=?
                )
        inner join
            WorkMode workmode3_
                on workbymode2_.workMode=workmode3_.code
        left outer join
            WorkShiftStatistics_workByMode workbymode4_
                on workshifts0_.id=workbymode4_.statistics_id
                and (
                    workbymode4_.workMode=?
                )
        inner join
            WorkMode workmode5_
                on workbymode4_.workMode=workmode5_.code
        where
            (
                workbymode2_.count is not null
                or workbymode4_.count is not null
            )

If you need to take the issue somewhere, I could try to set up a test project but probably not this week...

the output sql

Could you please post the JPQL?
The SQL is not our output in this case.
I think I see what is wrong (we might not reuse generated aliases correctly?), but to confirm my theory I need the JPQL that we generate.

This is the log output, is this what you need?

2016-08-25 10:43:41,589 [http-nio-8080-exec-3] DEBUG com.querydsl.jpa.hibernate.HibernateQuery - select count(workShiftStatistics) from WorkShiftStatistics workShiftStatistics   left join workShiftStatistics.user as user where (?1 member of workShiftStatistics.workModes or ?2 member of workShiftStatistics.workModes)
130476 [WARN] deprecation: HHH90000016: Found use of deprecated 'collection property' syntax in HQL/JPQL query [null.elements]; use collection function syntax instead [elements(null)].
130479 [WARN] deprecation: HHH90000016: Found use of deprecated 'collection property' syntax in HQL/JPQL query [null.elements]; use collection function syntax instead [elements(null)].
2016-08-25 10:43:41,626 [http-nio-8080-exec-3] DEBUG com.querydsl.jpa.hibernate.HibernateQuery - select workShiftStatistics.id, user.id, user.fullName, workShiftStatistics.workStartDate, workShiftStatistics.workEndDate, workShiftStatistics.netWorkTime, workShiftStatistics.active, workShiftStatistics.deviceId, workShiftStatistics.lastStartedWorkId, workShiftStatistics.lastStartedWorkType, workShiftStatistics.pickCount, workShiftStatistics.totalWeight, workShiftStatistics.lastStartedWorkDate, workShiftStatistics.lastFinishedWorkDate, workShiftStatistics.lastMessage, workShiftStatistics.lastMessageDate, workShiftStatistics.lastWorkMode.code, '' as workByMode from WorkShiftStatistics workShiftStatistics   left join workShiftStatistics.user as user where (?1 member of workShiftStatistics.workModes or ?2 member of workShiftStatistics.workModes) order by workShiftStatistics.lastMessageDate desc

Querydsl doesn't provide DSL alternatives for the JPQL KEY operator, but you can use it manually like this instead of map.containsKey(...):
query.from(myEntity)
         .leftJoin(myEntity.workByMode, workByMode)
         .on(Expressions.operation(WorkMode.class, JPQLOps.KEY, workByMode).in(key1, key2))
        .where(...)


The disable internal FXAA exeFS 1.1.2 mod @nsobid is wrong

It currently reads:
@nsobid-9A10ED9435C06733DA597D8094D9000AB5D3EE6
It should read:
@nsobid-9A10ED9435C06733DA597D8094D9000AB5D3EE60
(it's missing the zero at the end)

the zeroes at the end doesnt matter


Implemented file export progress dialog (#111)

Added feature for issue #111 - a progress dialog that displays and refreshes while extracting data from the package. It is also cancelable in case the user doesn't want to wait for it to finish.

Thanks, this looks great. One thing though, either the progress window should be closed on end, or the text should be changed to say its finished.

Done!


[SPARK-26917][SQL] Further reduce locks in CacheManager

What changes were proposed in this pull request?
Further load increases in our production environment have shown that even the read locks can cause some contention, since they contain a mechanism that turns a read lock into an exclusive lock if a writer has been starved out.  This PR reduces the potential for lock contention even further than https://github.com/apache/spark/pull/23833.  Additionally, it uses more idiomatic scala than the previous implementation.
@cloud-fan & @gatorsmile This is a relatively minor improvement to the previous CacheManager changes.  At this point, I think we finally are doing the minimum possible amount of locking.
How was this patch tested?
Has been tested on a live system where the blocking was causing major issues and it is working well.
CacheManager has no explicit unit test but is used in many places internally as part of the SharedState.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

looks reasonable

Good question. I had the same question in my mind. I just think that it is ok since this is shallow copy for a list of objects with two fields.
It would be good if there is statistics regarding # of objects in multiple scenario.

I don't think this will be a significant allocation as it's a shallow copy. At smaller scale, it shouldn't matter much. I'd take the win at larger scale

ok to test

shall we use some "copy-on-write" thread-safe collections instead of doing clone manually?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8737/
Test PASSed.

Test build #103294 has started for PR 24028 at commit 27ab9f0.

Test build #103294 has finished for PR 24028 at commit 27ab9f0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103294/
Test FAILed.

retest this please

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8746/
Test PASSed.

Merged build finished. Test PASSed.

Test build #103304 has started for PR 24028 at commit 27ab9f0.

Test build #103304 has finished for PR 24028 at commit 27ab9f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103304/
Test PASSed.

In response to @cloud-fan I updated this PR to stored the cached data in a scala immutable IndexedSeq.
As a result of this, there is no longer a need for read locks at all.  cachedData is changed to a var and all we need is a write lock for things that want to update the var.   From a code clarity perspective I think this solution is cleaner.  In addition to removing read locks, it removes several java to scala collection conversions.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8779/
Test PASSed.

Test build #103344 has started for PR 24028 at commit 1d6f84a.

Test build #103344 has finished for PR 24028 at commit 1d6f84a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103344/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8800/
Test PASSed.

Test build #103375 has started for PR 24028 at commit a5977f0.

Test build #103375 has finished for PR 24028 at commit a5977f0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103375/
Test FAILed.

Oh, looks like the problem is using tabs rather than spaces for indent

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8810/
Test PASSed.

Test build #103384 has started for PR 24028 at commit 1de029c.

Test build #103384 has finished for PR 24028 at commit 1de029c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103384/
Test PASSed.

Test build #103400 has finished for PR 24028 at commit 755d484.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103400/
Test PASSed.

Test build #103443 has started for PR 24028 at commit d402533.

Test build #103443 has finished for PR 24028 at commit d402533.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103443/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8864/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8868/
Test PASSed.

Test build #103448 has started for PR 24028 at commit 1d693af.

It looks like the build for the latest commit got messed up.  It says here that it is still going, but on the Jenkins server it looks like it completed.

LGTM, pending jenkins

It seems the Jenkins run stopped in the middle?

retest this please

Test build #103478 has started for PR 24028 at commit 1d693af.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8896/
Test PASSed.

Test build #103478 has finished for PR 24028 at commit 1d693af.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103478/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8900/
Test PASSed.

Test build #103485 has started for PR 24028 at commit 1d693af.

Test build #103485 has finished for PR 24028 at commit 1d693af.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103485/
Test PASSed.

Test build #103504 has started for PR 24028 at commit 1bb2511.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8919/
Test PASSed.

Test build #103504 has finished for PR 24028 at commit 1bb2511.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

thanks, merging to master!


Radio is still scanning old memory locations if a ScanList is enabled for that memory, but the memory channel isn't configured (i.e. deleted)

Discussed in https://github.com/AubsUK/uv-k5-firmware-custom/discussions/21

Originally posted by 121tech April 11, 2024
Chose your firmware fork specifically for the multiple scanlist.
I'm pretty sure when I installed v0.02 that if I placed memory channels in a scanlist bank, enabled multiple scanlist banks, then set the radio to scan then all the active scanlist banks got swept.
ISSUE:
Just updated to v0.06 and, maybe I'm remembering wrong as I appear to have lost the ability to scan multiple scanlist banks. The radio will now only scan the lowest enabled scanlist number and refuses to scan any other enabled scanlist bank. The screen displays the stars and dashes but the scan ignores the star enabled scanlist.
I tried reflashing back to v0.02 but it has the same new behavior. Reflashed back on v0.06, tried to perform a factory reset (hold PTT + SIDE BUTTON 1 while turning on the radio, after 'RELEASE ALL KEYS' navigate to Menu 77 Reset ), now 68.68 under v.006, but the scan function remains only able to scan the lowest enabled scanbank.
Radio is still scanning old memory locations if a ScanList is enabled in that location, but the memory channel isn't configured (i.e. deleted)

Fixed in v00.07

I've also fixed this in the latest CHIRP driver too, just download it from https://github.com/AubsUK/uvk5-chirp-driver/blob/main/uvk5_Aubs.py


Retry getting agent settings

Make agent code more resilient, e.g. to handle temporary network issues
#129348495
Signed-off-by: Felix Riegger<EMAIL_ADDRESS>

In general all units and integration tests are green.
However, we see one integration test failing on our local machine It labels the raw ephemeral paths for unpartitioned disks. This one was failing already before we introduced our changes. On the pipeline this commit runs green. Any idea why this could be failing in our case? Maybe a Mac thing?

Story to investigate the flaky/failing raw ephemeral paths test: https://www.pivotaltracker.com/story/show/130333367

Merge story: https://www.pivotaltracker.com/story/show/132002823

Thanks @dpb587-pivotal !


Add localisation

Today string (should probably be overridable by user)
Month short names in MonthView
Week single name characters

Closed via https://github.com/noona-hq/react-native-week-month-date-picker/commit/9256c10a8614f3141f90e3fb523bc2467d6a4bc7


New gas mask sprite in metal

This resolves #336

W00t @oyvindjam! This looks super cool. Thank you! 👍


Migrate module from rtyler/jenkins to puppet/jenkins

This comes from https://github.com/voxpupuli/puppet-jenkins/issues/1089.
https://forge.puppet.com/modules/rtyler/jenkins has been migrated to https://github.com/jenkinsci/puppet-jenkins which has since transferred to https://github.com/voxpupuli/puppet-jenkins.
https://forge.puppet.com/modules/rtyler/jenkins is still an approved module, but without a release since 2016. I don't think there ever was a release form the Jenkins namespace.
I'd like to deprecate https://forge.puppet.com/modules/rtyler/jenkins and recommend migration to https://forge.puppetlabs.com/modules/puppet/jenkins. I'm neutral about moving the approval, but obviously the approval should be removed from the deprecated module.

Deprecated https://forge.puppet.com/modules/rtyler/jenkins against https://forge.puppetlabs.com/modules/puppet/jenkins, hence closing the same.


CI failing silently

Actual Behavior
CI has been failing since 12.03.2020, when the last successful marketplace happened.
Versions were bumped and committed to the repo still.

Closed with https://github.com/prisma-labs/vscode-graphql/pull/187


Haddock can't process types-compat-0.1.0

from apiary/#15
Haddock from GHC 7.8.4 fails with an internal error trying to process this package:

    Haddock coverage:
      67% (  2 /  3) in 'Data.Proxy.Compat'
      86% ( 19 / 22) in 'Data.Typeable.Compat'
      93% ( 13 / 14) in 'GHC.TypeLits.Compat'
    Warning: Data.Proxy.Compat: could not find link destinations for:
        GHC.Generics.D1Proxy GHC.Generics.C1_0Proxy
    Warning: Data.Typeable.Compat: could not find link destinations for:
        GHC.Arr.STArray GHC.Arr.Array GHC.Generics.D1Proxy GHC.Generics.C1_0Proxy
    haddock: internal error: synifyTyCon: type/data family confusion

It seems like haddock's bug.
Please see haddock/#310


Wrong version in Maven repo

Hi,
Version 1.2.8 I got from mvnRepository seems to be an old one that doesn't support v4 gitlab's api. On the other hand, the version I get when building the project myself (through gradle) is 1.2.7-SNAPSHOT as it is defined in build.gradle.
Am I doing anything wrong?
TIA

Forgot to mention that the version built by myself supports v4 api

You're not doing anything wrong. I don't think they've published v4 support yet since it appears like there is some active development happening around the changes in the API versions. Publishing locally works because master contains some code for the v4 API, but is likely still under active development. I'm sure once the release for the v4 API is ready, they will publish it to maven (hopefully with a new major or minor version number).
From the read of the API migration docs, it looks like they won't be completely removing support for v3 until mid to end of August, so there is still a little bit of time to get this published.
https://docs.gitlab.com/ce/api/v3_to_v4.html

yea, that makes sense. What confused me is the version 1.2.7-SNAPSHOT I got from master, which is behind from what they got published in maven repo.
Anyhow, thanks for you input.

That is odd. I know that as a part of the release process, maven will bump that to 1.2.9-SNAPSHOT so I wonder if something didn't get merged back or if they are just manually managing those version numbers (I suspect not).
Looks like the gradle file has the same thing.

Releases are done with Maven. The version is in the pom. I don't know why there is also a build.gradle...
See #222 for regarding release with v4 support.
Also, always use Maven Central to look for libraries. MvnRepository is rarely up to date and contains a lot of advertising.


Add a test for assert_array_equal(strides_check=True)

Add a missing test for assert_array_equal(strides_check=True) introduced in #2150.

Jenkins, test this please

Jenkins CI test (for commit 58403005cae9d2ab9d8771b470381c48f66e375c, target branch master) succeeded!

LGTM


Ensure urljoin behaviour is correct

Upstream bugfix for https://github.com/mkdocstrings/mkdocstrings/pull/443/
I have attempted to write the necessary tests but the layout of the library is interfering a bit:
from mkdocs_gen_files import plugin

Traceback (most recent call last):
  File "/home/davfsa/coding/mkdocs-gen-files/tests/test_plugin.py", line 4, in <module>
    from mkdocs_gen_files import plugin
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/davfsa/coding/mkdocs-gen-files/mkdocs_gen_files/__init__.py", line 20, in __getattr__
    return getattr(FilesEditor.current(), name)
  File "/home/davfsa/coding/mkdocs-gen-files/mkdocs_gen_files/editor.py", line 94, in current
    config = load_config("mkdocs.yml")
  File "/home/davfsa/coding/mkdocs-gen-files/.venv/lib/python3.10/site-packages/mkdocs/config/base.py", line 236, in load_config
    raise exceptions.Abort(
mkdocs.exceptions.Abort: Aborted with 3 Configuration Errors!

Not sure what the best way to fix this would be

Thanks!

I finally made a release with this. https://github.com/oprypin/mkdocs-gen-files/releases/tag/v0.3.5


incus-unwrapped, lxd-to-incus: 0.2 -> 0.3

Description of changes
https://github.com/lxc/incus/releases/tag/v0.3.0
https://discuss.linuxcontainers.org/t/incus-0-3-has-been-released/18351

incus-unwrapped: 0.2 -> 0.3
lxd-to-incus: 0.2 -> 0.3

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Priorities
Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 270416 run on x86_64-linux 1

3 packages built:

incus
incus-unwrapped
lxd-to-incus

Result of nixpkgs-review pr 270416 run on aarch64-linux 1

3 packages built:

incus
incus-unwrapped
lxd-to-incus


add

Docs Build status updates of commit c7032b9:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details
Dependent Files

publis-test/reviewing-specs.md
:x:Error

Details

publis-test/reviewing-specs.md

Line 52: [Error] Cannot resolve './not_existed.md' relative to 'publis-test/reviewing-specs.md'.
Line 4: [Warning] Invalid value for author: 'misampso' is not a valid GitHub ID

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


gcstar 1.7.1

Motivation for this change
Things done
Built my first nix package from scratch.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

sorry for git messup I'll close the PR and open a new one


Unable to use mirrord on Java Service with connection to Kafka and MongoDB

Bug Description
Hello
I tried to use mirrord on a Java (Spring Boot) service which is connected to Kafka and MongoDB.
The Service is using Spring Kafka Streams and the regular MongoDB Client.
Though, env mirroring and dns resolving seems to work, I get Network is unreachable Exceptions when starting up the Kafka Client and also MongoDB Client is unable to create the connection.
My configuration File is:
{
"feature": {
"network": {
"incoming": "mirror",
"outgoing": {
"ignore_localhost": true,
},
"dns": true
},
"fs": "read",
"env": true
},
"target": {
"path": {
"deployment": "$SERVICE_NAME"
},
"namespace": "$NAMESPACE"
},
"agent": {
"log_level": "mirrord=info,warn"
},
"kube_context": "$CONTEXT",
"telemetry": false
}
Exceptions appear independently of using mirrord as standalone or plugin in IntelliJ
Steps to Reproduce

mirrord exec -f .mirrord/mirrord.json java -- -jar build/libs/service.jar

Backtrace
java.net.SocketException: Network is unreachable
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:579)
	at java.base/sun.nio.ch.Net.connect(Net.java:586)
	at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853)
	at org.apache.kafka.common.network.Selector.doConnect(Selector.java:277)
	at org.apache.kafka.common.network.Selector.connect(Selector.java:255)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:990)
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:498)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:255)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.coordinatorUnknownAndUnready(ConsumerCoordinator.java:460)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:488)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
	at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:980)
	at org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:933)
	at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:751)
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:604)
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:576)

Relevant Logs
No response
Your operating system and version
Mac OS Sonoma 14.3, Apple Silicone M1 Java 17
Local process
java,  Version 17
Local process version
No response
Additional Info
No response

Hi,
Sorry about the troubles. Can you please re-run with trace logs ?
MIRRORD_PROGRESS_MODE=off RUST_LOG=mirrord=trace mirrord exec -f .mirrord/mirrord.json java -- -jar build/libs/service.jar
Please make sure to take out any sensitive information from the logs before attaching.
If you'd like, we can also schedule a screen share session using this link.

Hello, sorry for answering that late, but here a the output of the command.
Seems to be a permission problem
024-02-05T13:42:43.140986Z  INFO mirrord: Launching "java" with arguments ["-jar", "build/libs/service.jar"]
2024-02-05T13:42:47.180654Z TRACE start:create_and_connect:create_session:fetch_operator: mirrord_operator::client: return=Err(KubeError { error: HyperError(hyper::Error(Connect, Custom { kind: Other, error: Os { code: 54, kind: ConnectionReset, message: "Connection reset by peer" } })), operation: FindingOperator })
2024-02-05T13:42:47.181049Z TRACE start:create_and_connect: mirrord::connection: Failed to connect to the operator, probably due to RBAC: finding operator failed: HyperError: error trying to connect: Connection reset by peer (os error 54)
2024-02-05T13:42:51.222277Z DEBUG start:create_and_connect: mirrord::connection: couldn't determine OpenShift: mirrord-layer: Kube failed with error `HyperError: error trying to connect: Connection reset by peer (os error 54)`!
Error:   × Create agent failed. KubeError(
  │     HyperError(
  │         hyper::Error(
  │             Connect,
  │             Custom {
  │                 kind: Other,
  │                 error: Os {
  │                     code: 54,
  │                     kind: ConnectionReset,
  │                     message: "Connection reset by peer",
  │                 },
  │             },
  │         ),
  │     ),
  │ )

Thank's for your answer.
When trying today, to gather the trace logs, it worked like a charm.
Though I really don't know why :-)
I'll close this issue !


use configured git author in scaffolder actions

Overview
this PR addresses #6217 by allowing the git author info used by the scaffolder to be configured instead of always using the hardcoded values.
Approach
this is currently accomplished by passing Config through to createBuiltinActions and subsequently to each relevant create*Action (e.g. createPublishGithubAction).
we considered passing only the relevant configuration information (gitAuthorInfo?: { name?: string; email?: string }) instead of the full Config object. while this would be more lightweight, it would mean that future additional configuration needs would require modifying this object everywhere it's needed.
a third option would be to create a GitConfig class (similar to ScmIntegrations) to parse and hold this information as well as potential future related configuration data.
we'd appreciate input on which approach other folks would consider most consistent/appropriate.
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[x] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

I took a peak and I think the "best" option is to keep Config to have flexibility in the future for when we might add new options. One alternative could be to make config optional but I could see that being confusing if you configure the author and forget to pass the config object if you have default and custom actions.
I vote for making this a breaking change in the same release as #6123. Please add a changeset with instructions for how to migrate

Do we want to have the git parent key? Maybe call it scaffolder.defaultAuthor or something like that since we intend to pass on the actual end user credentials as well in the future to override these when possible? @jhaals @benjdlambert
A small conflict has crept in as well.

good shout @freben. considering it will be inherited from the user that actioned the job it's nicer that way I think.

@chaseajen looks like there's some conflicts here, do you mind fixing them up?

Also this will need a changeset describing the changes and a migration sample of what users now need to do 👍

resolved conflicts and added a changeset ✅

Just changed this to a minor change, but this looks good now I think.


➡ Migrate core package 'autoflow' into ./packages

Description of the Change
This PR migrates the core autoflow package into atom/atom.  See issue #17833 for more information.

Same ol' Linux flake on VSTS, I'll look into that after merging this.


[CI:BUILD] RPM: cleanup spec and fix eln builds

remove unnecessary conditionals
fix ELN builds on copr environments

Note: The ELN build fix is actually handled on the COPR side by enabling the local repo. It's usually disabled by default on copr and enabled by default on koji, sometimes leading to inconsistent results.

good to go. PTAL

@Luap99 same thing as netavark PR. no worries if release commit merged already.


Unable to migrate subfolder containing .git folder

If a project/directory in RTC contains .git folder, it won't be migrated to git.
I occasionally added .git folder when initially shared my project in RTC. Now when I have to migrate it to GIT the migration plug-in leaves the project folder empty.
There should be a way to override the current behaviour since otherwise it's impossible to import that project.

@akoptelov what do you suggest as a possible way? Try to ignore .git folders that come from RTC somehow? What with contents of such a folder?

@reinhapa Adding the .git to .gitignore works for plain git, but doesn't for JGit version that the rtc2gitcli is using. Still there's a bug report and code targeting it is already in the stable JGit since 4.3.
As I see it, there can be two scenarios. One is that there is a real GIT repository in RTC, and in this case current behaviour is perfectly OK since that repo should be migrated separately. And the second one is when the repository metadata got to the RTC by mistake and doesn't have any value and thus .git should be ignored completely and all files should be migrated normally.
The solution would be to adopt JGit 4.3 (or newer?) and then implement possibility to specify additional git options (core/dirNoGitLinks boolean flag does the trick). I guess we can introduce general way to specify them via properties.
If you're ok with this, I can create a pull request for it.

@akoptelov yes please, due the fact, that we have migrated all our code now and do no longer have an RTC repo here, I have no chance to reproduce nor test this...

@reinhapa Closing this issue.


Validators could use more tests

Validators have low coverage.  Lets fix that.
molecule/validators.py                             69     49    29%

Look's like the coverage isn't too too bad. Just the trailing that really needs most of the attention.
molecule/verifier/base.py                         7      1    86%
molecule/verifier/serverspec.py                  33      8    76%
molecule/verifier/testinfra.py                   33     11    67%
molecule/verifier/trailing.py                    54     33    39%

Yeah, now that they were broken out, they got some extra coverage.  The tree crawling going to be tricky.

Going to close, we have bigger fish to fry, and is no longer much of an issue after the verifier refactoring.


Editing files collection is very slow when you have a lot of entries

Describe the bug
Editing files collections is very slow, by making them almost unusable when you have a lot of entries.
To Reproduce
See attached zipped file.json sample containing 1000 items. Each item has just two fields:

name
id

Define a files collection as follow:
collections:
  - label: Files Collections
    name: files_collections
    files:
      - label: Items
        name: items
        file: data/file.json
        fields:
          - label: Items
            label_singular: Item
            name: items
            widget: list
            fields:
              - label: Name
                name: name
                widget: string
              - label: ID
                name: id
                widget: string

Listing is very slow (about 5 seconds). If you open the developer tool and reload the item list you'll see few violations. But the worst performance it happens when you add a new entry by using the Add item button:

it takes about 3/4 seconds before being able to scroll all the way down to the end of the list;
it takes about 6 seconds from the time you start writing in a field until you receive the feedback, making the editing of an entry unusable.

Expected behavior
The interaction with a file collection should be much more responsive, even with a lot of items
Applicable Versions:

Netlify CMS version: 2.12.15
Git provider: git-gateway/github
OS: ubuntu 18.04
Browser version: Google Chrome 83.0.4103.106

CMS configuration
See Above
file.zip

I think this issue is the same as #2009 and #3415

Closing in favour of https://github.com/netlify/netlify-cms/issues/3415


Meeting Minutes - Milestone 2

Tues. Apr 13 - 2pm - 4pm PST(during lab)

walked through Exercises 1-5 together as a group with one person screen sharing.
confirmed all members have access to AWS EC2 instance and JupyterHub login

Next meeting 11am PST on Friday, April 16 to complete Q6 together and submit.  No prep required before meeting.

Friday, Apr 16 meeting:
As a group, with screen sharing:

completed data wrangling & merging table (step 6)
downloaded ipynb and added Milestone markdown steps & screenshots embedded into file.
uploaded screenshot images and milestone2.ipynb to this repo.
submitted milestone 2 link as a group


"Error performing HTTP request"

Hello. I am not a strong in Linux admin so apologies in advance if I haven't provided enough information. I am attempting to install Chevereto on an Ubuntu Server 20.04 LTS.

Apache: <IP_ADDRESS>
PHP: 7.4.3
MariaDB 10.4

I confirmed that Apache and PHP were working and created a database. I am able to bring up the Installer page for the free edition and to run it.  The installer fails, but reports the following.
Server <IP_ADDRESS> PHP 7.4.3
Software has been set to: chevereto-free
Database cheveretodb OK
Detecting existing cPanel .htaccess handlers
No .htaccess found
Downloading latest chevereto-free release
Downloaded chevereto-pkg-a4f202c7.zip (4.7 MB @3.63MB/s.)
Extracting chevereto-pkg-a4f202c7.zip
Extraction completeted (424 files in 0.23s)
Creating app/settings.php file
Settings file OK
Performing system setup
Error performing HTTP request
Process aborted

Is there some port or permission I may have missed that I need to enable on the server in order for the installer to continue?

Read the docs: https://chevereto.com/docs/installer


[WFCORE-1892] Run checkFailedTransformedBootOperations() in a batch t…

…o deal with caps/reqs
https://issues.jboss.org/browse/WFCORE-1892
We will see problems until https://github.com/wildfly/wildfly/pull/9351 has been merged

Core - Full Integration Build 3705 outcome was FAILURE using a merge of fa0531a5d3c37c609cf60534742e5272148be40d
Summary: Compilation error: Compiler (new) Build time: 00:00:55

Full integration - Windows Build 2149 outcome was FAILURE using a merge of fa0531a5d3c37c609cf60534742e5272148be40d
Summary: Compilation error: Compiler (new) Build time: 00:01:43

Linux Build 4751 outcome was FAILURE using a merge of fa0531a5d3c37c609cf60534742e5272148be40d
Summary: Tests passed: 1271, ignored: 5; compilation error: Compiler (new) Build time: 00:01:59

Windows Build 4066 outcome was FAILURE using a merge of fa0531a5d3c37c609cf60534742e5272148be40d
Summary: Tests passed: 1271, ignored: 5; compilation error: Compiler (new) Build time: 00:02:34


(2106) Add fields to qualifications page

This adds four new optional fields to the qualifications page:

Recognition of qualifications in the UK (string)
More about recognition within the UK (url)
Recognition of qualifications from other countries (string)
More about recognition in other countries (url)

If the profession covers all UK nations, the first two questions are hidden.
I've also rearranged the fields to match the order in the current prototype.
Screenshots

/rebase


Fix/strings and input

Also, this PR fixes a bug with quotation marks in strings being stored as \" rather than ".


Documentation

Create proper documentation
a.	Start by converting the text in https://arxiv.org/abs/2309.04841 into docs
E.g. similar to https://openqaoa.entropicalabs.com/ (uses MKDocs) or https://xgboost.readthedocs.io/en/stable/
b.	Study code and ask specific questions to code developers to add documentation for pieces not documented in the arXiv paper

(Moving text over from #25 )
At a high level, the workflow we are hoping to enable with QOKit is as follows:

Use get_qaoa_maxcut_objective to find good parameter beta gamma (used fixed? Optimize?) for your input graph
Use these parameter to generate a QAOA circuit to run on hardware (qokit/qaoa_circuit_maxcut.py)

The same flow should work for other problems (portfolio optimization, LABS, ...)


Update mockito-scala, ... to 1.16.21

Updates

org.mockito:mockito-scala
org.mockito:mockito-scala-scalatest

from 1.16.15 to 1.16.21.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.mockito" } ]

labels: test-library-update, semver-patch

Superseded by #788.


TextInput - selection prop is not set on component creation

Summary:
Problem:
selection prop is not being set on component creation.
Not quite sure which RN version this issue was introduced but fixing it on latest code.
Use playground for testing (refer to following diff)
Proposed Solution:
Added notes in comments but viewCommands.setTextAndSelection() is called only on text or selection update which relies on comparing data with lastNativeSelection. Problem is that lastNativeSelection is initially set to the props value that is passed in so does not send the command on component creation.
So assign a default selection value of {start: -1, end: -1} so it can be set on component creation.
Changelog:
[General][Fixed] - 'selectionprop inTextInput` was not being applied at component creation
Differential Revision: D56911712

This PR fixes issue reported here: https://github.com/facebook/react-native/issues/44171


Feat: Allow user to close numeric keyboard on iOS

Summary
As described in #402 , the user is unable to close the numeric keyboard on iOS. On some occasions this causes a bad user experience.
This PR fixes this by adding a press event handler on the screens that exhibit a numeric keyboard.
Acceptance Criteria

The user should be able to close the numeric keyboard on iOS
Fixes #402

[!TIP]
In total, just a few lines had actual text modified. The rest of the modified lines were only due to indentation adjustments.
To better visualize the actual changes I suggest using the "Hide whitespace" feature on GitHub, as it highlights only the words that were changed on this file, and not the indentation adjustments that were necessary:

Demonstration
Although the video does not exhibit the finger press locations, it's possible to understand the overall interaction: by pressing anywhere on the screen other than the numeric input, the user is able to close the numeric keyboard.
https://github.com/HathorNetwork/hathor-wallet-mobile/assets/1299409/3bb39761-7713-41dc-a35d-4c77dcfcde29
Security Checklist

[X] Make sure you do not include new dependencies in the project unless strictly necessary and do not include dev-dependencies as production ones. More dependencies increase the possibility of one of them being hijacked and affecting us.

[non-blocking] Why is this detrimental for the experience? This screen has only one purpose: informe the value.

@alexruzenhack This was causing issues when, for some reason, the keyboard got in the way of the information being shown on screen, such as a button.
The main solution to this problem was implemented on #325 , but the option to hide the keyboard was still to be added.