BLE GATT server doesn't read/write value

Environment

Development Kit:      [none]
Kit version (for WroverKit/PicoKit/DevKitC): []
Module or chip used:  [ESP32-WROVER]
IDF version (run git describe --tags to find it): v3.1-151-g7063cdecd
Build System:         [Make]
Compiler version (run xtensa-esp32-elf-gcc --version to find it): 1.22.0-80-g6c4433a5
Operating System:     [Windows]
Power Supply:         [external 3.3V]

Problem Description
BLE GATT server does not read or write my global array when characteristic is defined with ESP_GATT_AUTO_RSP.
Expected Behavior
The global array that I provided via pointer to the stack on initialization should be read from and written to by the stack when the characteristic is defined with ESP_GATT_AUTO_RSP.
Actual Behavior
I can read and write the characteristic via BLE, but it seems my global array is never changed.
So it seems there is some internal buffer in the BLE stack storing the value because I can read back what I wrote via BLE.
It seems the global array that I passed to the stack on creation is only used to initialize the internal value.
Steps to reproduce

Add a few debug statements to the example at esp-idf\examples\bluetooth\gatt_server_service_table\main\gatts_table_creat_demo.c
(or the better organized example project here: https://github.com/eagi223/esp-idf_Bluetooth_Multi-Service )
...
When I first read the characteristic, it returns what I have initialized the global to. I read [11 22 33 44], just as expected.
Then I write the value via BLE to 0x1234.
I can then read via BLE and it returns [12 34] as expected.
But my global array has not been changed.
Also, if I change my global array value via other means (not via BLE), the value I read via BLE is not changed.

Code to reproduce this issue
We have some characteristics defined using ESP_GATT_AUTO_RSP, and I supplied a pointer to the value when creating the gatt table.
The value is stored in a global uint8_t array.
static const uint8_t char_value[4]                 = {0x11, 0x22, 0x33, 0x44}; // global array to store char value
...
const esp_gatts_attr_db_t serv_gatt_db[SERV_NUM_ATTR] =
{
...
    /* Characteristic Value */
    [IDX_CHAR_VAL_A] =
    {{ESP_GATT_AUTO_RSP}, {ESP_UUID_LEN_16, (uint8_t *)&GATTS_CHAR_UUID_TEST_A, ESP_GATT_PERM_READ | ESP_GATT_PERM_WRITE,
      GATTS_DEMO_CHAR_VAL_LEN_MAX, sizeof(char_value), (uint8_t *)char_value}},
...

I added some debug hints to print out the value of char_value[4] in gatts_profile_event_handler:
        case ESP_GATTS_READ_EVT:
            ESP_LOGI(GATTS_TABLE_TAG, "ESP_GATTS_READ_EVT char_value:");
            ESP_LOG_BUFFER_HEX_LEVEL(GATTS_TABLE_TAG, char_value, sizeof(char_value), ESP_LOG_INFO);
       	    break;
        case ESP_GATTS_WRITE_EVT:
...
            ESP_LOGI(GATTS_TABLE_TAG, "GATT_WRITE_EVT, char_value :");
            ESP_LOG_BUFFER_HEX_LEVEL(GATTS_TABLE_TAG, char_value, sizeof(char_value), ESP_LOG_INFO);
...
        case ESP_GATTS_EXEC_WRITE_EVT:
...
            ESP_LOGI(GATTS_TABLE_TAG, "ESP_GATTS_EXEC_WRITE_EVT, char_value :");
            ESP_LOG_BUFFER_HEX_LEVEL(GATTS_TABLE_TAG, char_value, sizeof(char_value), ESP_LOG_INFO);
...

Debug Logs
I (627) GATTS_TABLE_DEMO: create attribute table successfully, the number handle = 8
I (627) GATTS_TABLE_DEMO: SERVICE_START_EVT, status 0, service_handle 40
I (637) GATTS_TABLE_DEMO: advertising start successfully
I (236277) GATTS_TABLE_DEMO: ESP_GATTS_CONNECT_EVT, conn_id = 0
I (236277) GATTS_TABLE_DEMO: 64 1d c8 4e fe 1d
I (236957) GATTS_TABLE_DEMO: update connection params status = 0, min_int = 16,
max_int = 32,conn_int = 30,latency = 0, timeout = 400
I (237747) GATTS_TABLE_DEMO: update connection params status = 0, min_int = 0, m
ax_int = 0,conn_int = 6,latency = 0, timeout = 2000
I (237897) GATTS_TABLE_DEMO: update connection params status = 0, min_int = 0, m
ax_int = 0,conn_int = 30,latency = 0, timeout = 400
I (245967) GATTS_TABLE_DEMO: ESP_GATTS_READ_EVT char_value:
I (245967) GATTS_TABLE_DEMO: 11 22 33 44
I (264337) GATTS_TABLE_DEMO: GATT_WRITE_EVT, handle = 42, value len = 2, value :+1:
I (264337) GATTS_TABLE_DEMO: 12 34
I (264337) GATTS_TABLE_DEMO: GATT_WRITE_EVT, char_value :
I (264347) GATTS_TABLE_DEMO: 11 22 33 44
I (277017) GATTS_TABLE_DEMO: ESP_GATTS_READ_EVT char_value:
I (277017) GATTS_TABLE_DEMO: 11 22 33 44

Other items if possible
will provide if requested
Questions:

How and when do I access the internal value?

If the BLE stack is supposed to write to my global array automatically, does it do so before or after triggering ESP_GATTS_WRITE_EVT (or ESP_GATTS_EXEC_WRITE_EVT)?

If the BLE stack is supposed to read from my global array automatically, does it do so before or after triggering ESP_GATTS_READ_EVT?

Where is the intended behavior documented?

Note: I realize that I could use ESP_GATT_RSP_BY_APP and write code to handle the read/write to my global array. But I am using long-write and long-read, because the size of array may be larger than MTU. i.e. static uint8_t char_value[128] I want to use ESP_GATT_AUTO_RSP because it seems that the BLE stack is (somehow) taking care of the chunking required for long-write and long-read automatically.
BTW: I asked this on esp32.com a week ago and got no response: https://www.esp32.com/viewtopic.php?f=13&t=9345 .  Also here's a relevant post: https://www.esp32.com/viewtopic.php?f=13&t=2459

Hi phatpaul :
If you set ESP_GATT_AUTO_RSP, the Bluetooth stack will have a buffer to store the initialized value of the char. When the master reads or writes the char, the buffer will be read or updated.  if you change your global array value via other means (not via BLE), please call esp_ble_gatts_set_attr_value() to update the buffer.

you can use esp_ble_gatts_get_attr_value() and esp_ble_gatts_set_attr_value() to  read and update the  internal buffer value.

Great thanks @Weijian-Espressif .  That's the function I was looking for.
But it doesn't seem well documented.  Perhaps add some notes that mention ESP_GATT_AUTO_RSP to the documentation of those functions?  And it would be helpful if the example made use of those.
Also, my other questions, which I will try to figure out experimentally:

Is the internal value updated before or after triggering ESP_GATTS_WRITE_EVT (or ESP_GATTS_EXEC_WRITE_EVT)?  (it would only be useful if the event triggers after the internal value is updated , so I can do something with the value)

Is the internal value send out BLE before or after triggering ESP_GATTS_READ_EVT?  (it would only be useful if event is triggered before sending, so I can change the value before it is sent)

So it looks like the answer is
* ESP_GATTS_ (EXEC_) WRITE_EVT triggers After a Write, which is good.

(I am using ESP_GATTS_EXEC_WRITE_EVT to receive long writes.  Seems I have to buffer the attr handle from one of the Prepare writes so I know which characteristic is being written to on a ESP_GATTS_EXEC_WRITE_EVT - as there is no reference to the handle given in that event?!)
But
* ESP_GATTS_READ_EVT triggers After a Read, which is not very helpful.
Is there another event which happens earlier during a read?  I want to have the opportunity to set the value before it is sent out.  (I suppose I could set the BLE internal char value whenever my value changes.  But it seems wasteful to frequently copy it to the BLE stack internal variable even if it is not being read by BLE)

@phatpaul,
you are right,
ESP_GATTS_ (EXEC_) WRITE_EVT/ ESP_GATTS_READ_EVT triggers After a Write/Read.
According to your needs, I think you can set it to ESP_GATT_RSP_BY_APP,  when ESP_GATTS_READ_EVT triggers, you send response with your global array value. when ESP_GATTS_WRITE_EVT triggers, you update your global array value. you  can refer to our gatt_server demo.

@Weijian-Espressif thanks for confirming.
I want to use ESP_GATT_AUTO_RSP because my characteristic length might be larger than MTU.  I didn't want to write my own code to handle the chunking for long-read and long-write.
I saw there is some code to implement long-write in the example, but I don't see any example for long-read.  (example_write_event_env() and example_exec_write_event_env())

Can you please provide an example implementation of long-read by App?

(I found this implementation in C++, but a C example would be helpful - https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/BLECharacteristic.cpp#L348 )

@phatpaul,
if you want to ESP_GATT_RSP_BY_APP, when you get ESP_GATTS_READ_EVT, you need to call esp_ble_gatts_send_response() to send your value, but the length cannot exceed 600 bytes.
Does this way meet your needs?

@Weijian-Espressif thanks, that worked for sending long value with ESP_GATT_RSP_BY_APP.

It seems it will not send any response if MTU is less than the length of my response.
I see a warning that says W (27859) BT_GATT: attribute value too long, to be truncated to 22
But I didn't receive a truncated 22-byte value, I got error 0x2: GATT READ NOT PERMIT.
So it seems I need to check MTU and truncate my response before calling
esp_ble_gatts_send_response()?

Will there be repeated ESP_GATTS_READ_EVT events for a single long-read?  Do I need to check param->read.is_long?  Like this:

if (!param->read.is_long) {
// If is.long is false then this is the first (or only) request to read data
...
    memcpy(rsp->attr_value.value, vol_db, 128);
    rsp->attr_value.len = 128;
...
}
if (param->read.need_rsp){ // needs external response?
    rsp.attr_value.handle = param->read.handle;
    esp_ble_gatts_send_response(gatts_if, param->read.conn_id, param->read.trans_id, status, &rsp);
}

If I use ESP_GATT_RSP_BY_APP it seems there is no internal variable in the stack?  When I try to use  esp_ble_gatts_get_attr_value() I get 0 length nothing.
So if use ESP_GATT_RSP_BY_APP, then I also have to use the external long-write functions
example_write_event_env() and example_exec_write_event_env(), correct?

Now I'm getting a better idea of the API.   I think the example should have a service with characteristics demonstrating the different possible implementations.

Long (600 byte) characteristic read/write/notify using ESP_GATT_AUTO_RSP
Long (600 byte) characteristic read/write/notify using ESP_GATT_RSP_BY_APP
Short (20 byte) characteristic read-only using ESP_GATT_AUTO_RSP
Short (20 byte) characteristic read/write/notify using ESP_GATT_AUTO_RSP
Short (20 byte) characteristic read/write/notify using ESP_GATT_RSP_BY_APP

In the example some characteristic should be able to

trigger an action like turn on/off an LED
return an analog value which is determined at the time of the read (i.e. a timestamp or analog input value)
notify on an external stimulus like an IO pin level change.

Thoughts?  Do you want me to help add these to the example?

@Weijian-Espressif let me take back my previous report that esp_ble_gatts_send_response() works for me.
It does not work to send a response which is longer than MTU.
I like the behavior that if I use ESP_GATT_AUTO_RSP, and MTU is still = 23, the long read just works.
But when I tried ESP_GATT_RSP_BY_APP and tried to read a 128B characteristic without requesting MTU increase, I just got a warning and no data
W (248683509) BT_GATT: attribute value too long, to be truncated to 22

So is it required to write some external code to allow sending > MTU if ESP_GATT_RSP_BY_APP?

@phatpaul , The length of a response  must less than MTU size.

Is the source code available for how the stack implements the long-read and chunking internally?
If so would you point it out please.  I'd rather not re-invent it if I have to do it at a higher level.

@phatpaul please refer.
// support read and long read
void gatts_proc_read(esp_gatt_if_t gatts_if, esp_ble_gatts_cb_param_t *param, uint8_t *p_rsp_v, uint16_t v_len)
{
    if(!param->read.need_rsp) {
        return;
    }
    uint16_t value_len = gatts_mtu - 1;
    if(v_len - param->read.offset < (gatts_mtu - 1)) {
        value_len = v_len - param->read.offset;
    }
    esp_gatt_rsp_t rsp;
    memset(&rsp, 0, sizeof(esp_gatt_rsp_t));
    rsp.attr_value.handle = param->read.handle;
    rsp.attr_value.len = value_len;
    rsp.attr_value.offset = param->read.offset;
    memcpy(rsp.attr_value.value, &p_rsp_v[param->read.offset], value_len);
    esp_ble_gatts_send_response(gatts_if, param->read.conn_id, param->read.trans_id, ESP_GATT_OK, &rsp);

}

case ESP_GATTS_READ_EVT: {
        ESP_LOGI(GATTS_TAG, "GATT_READ_EVT, conn_id %d, trans_id %d, handle %d offset %d\n", param->read.conn_id, param->read.trans_id, param->read.handle, param->read.offset);
        gatts_proc_read(gatts_if, param, read_rsp, sizeof(read_rsp));
        break;
    }

gatts_demo.c.zip

Thanks @Weijian-Espressif that helped a lot.
Here's my solution, since I want to trigger the APP read handler once when a long read is begun, and buffer the value from the APP read handler so it can be pushed out in chunks during subsequent read events.  I based it on your example plus the existing example_prepare_write_event_env() functions.
Would you please review it to look for memory leak etc?  Feel free to include it in the example.
typedef struct {
    uint8_t                 *prepare_buf;
    int                     prepare_len;
    uint16_t                handle;
} prepare_type_env_t;

static prepare_type_env_t prepare_write_env;
static prepare_type_env_t prepare_read_env;

...
// support read and long read
void gatts_proc_read(esp_gatt_if_t gatts_if, prepare_type_env_t *prepare_read_env, esp_ble_gatts_cb_param_t *param, uint8_t *p_rsp_v, uint16_t v_len)
{
	if(!param->read.need_rsp) {
		return;
	}
	uint16_t value_len = gatts_mtu - 1;
	if(v_len - param->read.offset < (gatts_mtu - 1)) { // read response will fit in one MTU?
		value_len = v_len - param->read.offset;
	}
	else if (param->read.offset == 0) // it's the start of a long read  (could also use param->read.is_long here?)
	{
		ESP_LOGI(TAG, "long read, handle = %d, value len = %d", param->read.handle, v_len);

		if (v_len > PREPARE_BUF_MAX_SIZE) {
			ESP_LOGE(TAG, "long read too long");
			return;
		}
		if (prepare_read_env->prepare_buf != NULL) {
			ESP_LOGW(TAG, "long read buffer not free");
			free(prepare_read_env->prepare_buf);
			prepare_read_env->prepare_buf = NULL;
		}

		prepare_read_env->prepare_buf = (uint8_t *)malloc(PREPARE_BUF_MAX_SIZE * sizeof(uint8_t));
		prepare_read_env->prepare_len = 0;
		if (prepare_read_env->prepare_buf == NULL) {
			ESP_LOGE(TAG, "long read no mem");
			return;
		}
		memcpy(prepare_read_env->prepare_buf, p_rsp_v, v_len);
		prepare_read_env->prepare_len = v_len;
		prepare_read_env->handle = param->read.handle;
	}
	esp_gatt_rsp_t rsp;
	memset(&rsp, 0, sizeof(esp_gatt_rsp_t));
	rsp.attr_value.handle = param->read.handle;
	rsp.attr_value.len = value_len;
	rsp.attr_value.offset = param->read.offset;
	memcpy(rsp.attr_value.value, &p_rsp_v[param->read.offset], value_len);
	esp_ble_gatts_send_response(gatts_if, param->read.conn_id, param->read.trans_id, ESP_GATT_OK, &rsp);

}

// continuation of read, use buffered value
void gatts_proc_long_read(esp_gatt_if_t gatts_if, prepare_type_env_t *prepare_read_env, esp_ble_gatts_cb_param_t *param)
{

	if (prepare_read_env->prepare_buf && (prepare_read_env->handle == param->read.handle)) // something buffered?
	{
		gatts_proc_read(gatts_if, prepare_read_env, param, prepare_read_env->prepare_buf, prepare_read_env->prepare_len);
		if(prepare_read_env->prepare_len - param->read.offset < (gatts_mtu - 1)) // last read?
		{
			free(prepare_read_env->prepare_buf);
			prepare_read_env->prepare_buf = NULL;
			prepare_read_env->prepare_len = 0;
			ESP_LOGI(TAG,"long_read ended");
		}
	}
	else
	{
		ESP_LOGE(TAG,"long_read not buffered");
	}
}

static void gatts_profile_event_handler(esp_gatts_cb_event_t event, esp_gatt_if_t gatts_if, esp_ble_gatts_cb_param_t *param)
{
...
        case ESP_GATTS_READ_EVT:
        {
            ESP_LOGI(TAG, "GATT_READ_EVT, conn_id %d, trans_id %d, handle %d offset %d", param->read.conn_id, param->read.trans_id, param->read.handle, param->read.offset);

			// Note that if char is defined with ESP_GATT_AUTO_RSP, then this event is triggered after the internal value has been sent.  So this event is not very useful.
			// Otherwise if it is ESP_GATT_RSP_BY_APP, then call helper function gatts_proc_read()
			if (!param->read.is_long) {
				// If is.long is false then this is the first (or only) request to read data
				int attrIndex;

				/*
				 *  Add calls to handle read events for each external service here
				 *  Template:
				 *  if( (attrIndex = getAttributeIndexBy#SERVICE#Handle(param->read.handle)) < #SERVICE MAX INDEX# )
				 *  {
				 *      ESP_LOGI(TAG, "#SERVICE# READ");
				 *      handle#SERVICE#ReadEvent(attrIndex, param, &rsp);
				 *  }
				 */

				if( (attrIndex = getAttributeIndexByMyHandle(param->read.handle)) < MY_SERV_NUM_ATTR )
				{
					//ESP_LOGI(TAG,"MY READ");
					handleMyReadEvent(attrIndex, &rsp);
				}

				/* END read handler calls for external services */
				gatts_proc_read(gatts_if, &prepare_read_env, param, rsp.attr_value.value, rsp.attr_value.len);
			}
			else  // a continuation of a long read.  Dont invoke the handle#SERVICE#ReadEvent again, just keep pumping out buffered data.
			{
				gatts_proc_long_read(gatts_if, &prepare_read_env, param);
			}
        }

//Handle any reads of characteristics here
void handleMyReadEvent(int attrIndex, esp_gatt_rsp_t* rsp)
{
	switch( attrIndex )
	{
	// Characteristic read values

	// Note that if char is defined with ESP_GATT_AUTO_RSP, this event is triggered AFTER the internal value has been sent.
	// So this event is not useful for setting the value with esp_ble_gatts_set_attr_value() - should do that somewhere else.
	// But if char is defined with ESP_GATT_RSP_BY_APP, then build the response here before it is sent:

	case MY_CHAR_VAL:
	{
		memcpy(rsp->attr_value.value, vol_db, VOL_DB_SIZE);
		rsp->attr_value.len = VOL_DB_SIZE;
		break;
	}

	}
}

@phatpaul, If there are no more questions, we will close this issue.

Thanks.  Close it.
FYI my example of long read/write working with any MTU is here.
https://github.com/eagi223/esp-idf_Bluetooth_Multi-Service/pull/6
On Sun, Mar 31, 2019, 10:35 PM weijian_Espressif<EMAIL_ADDRESS>wrote:

@phatpaul https://github.com/phatpaul, If there are no more questions,
we will close this issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/espressif/esp-idf/issues/3102#issuecomment-478415592,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADVVd0qUBEkSHXC9va9n9xIwFdxZv-bpks5vcXBugaJpZM4bScm2
.

@phatpaul Thanks for reporting the issue. Feel free to reopen if the issue still exists. Thanks.


Setup Restcontroller for Categories

This is finished in branch REST-API. Just need to pull when the rest is finished

Just remember these needs to have tests


modify: jib config for new shared services

JU-271

CI: Security warnings found!

CI: No security warnings found


Cannot start a runtime from within a runtime. This happens because a function (like block_on) attempted to block the current thread while the thread is being used to drive asynchronous tasks.

CHAIN_ID=11155111  RPC_URL=... PRIVATE_KEY=...CONTRACT_ADDRESS=... TENDERMINT_RPC_URL=<IP_ADDRESS>:26657  cargo run --bin operator

Hey! If you follow the instructions in the README this should now be fixed 🙂, thanks for opening the issue. Feel free to open another one if you run into anything else.
https://github.com/succinctlabs/sp1-tendermint-example?tab=readme-ov-file#run-tendermint-operator-end-to-end


Enable custom actions on existing claims or bundles

After reading this issue comment I took a look at how we could make it possible to run custom actions on bundles that duffle has built but not installed or bundles it does not know about. Since running a custom action currently requires a claim, the easiest thing to do was create a claim if one did not exist.
This PR allows duffle run to perform custom actions in the following scenarios

When the claim exists:
users can run duffle run someCustomAction--claim existingClaim and duffle will reference the claim store in order to perform the custom action.

When the claim does not exist but duffle knows about the bundle:
users can run duffle run someCustomAction --bundle someBundle --claim myNewClaim and duffle will look in the store to perform the custom action. It will also create a claim if the action is a modifying one.

When the claim doe not exist and duffle does not know about the bundle:
users can run duffle run someCustomAction --bundle-is-file some-bundle.json --claim myNewClaim and duffle will use the bundle definition to run the custom action. It will also create a claim if the action is a modifying one.

Let me know what you all think of this functionality. I'm open to feedback and restructuring the design of this cli command.
Related Issues:

https://github.com/deislabs/duffle/issues/629
https://github.com/deislabs/duffle/issues/830

Hi @radu-matei @glyn @jeremyrickard @carolynvs! Would be great to get some 👀 on this one when you get a chance.

Thanks for taking a look @glyn!

The approach of creating new claims just to run custom actions seems to change that and might cause issues.

I was going back and forth on this. Attaching a bundle definition to a new claim was the easiest way to maintain the current interface for creating a driver operation. But I could have also implemented this with a temporary claim that served the purpose of creating the operation and then not persist the claim in duffle's store. This would maintain the constraint that claims should only be created on install. I ended up keeping the claim around because it was the only way for a user to keep track of which action had been run and its status.
The delete implications are valid concerns and I would be happy to make changes that didn't persist the claim in the duffle store. I think that's the right thing to do since the spec specifically defines claims as installed bundles. How would you feel about the PR if I made that change?

Also, it seems slightly odd that modifying custom actions should save their claims, but non-modifying actions shouldn't as that makes the lifecycle of claims even more complicated to reason about.

This was just to maintain the current logic here.

Thanks for taking a look @glyn!

The approach of creating new claims just to run custom actions seems to change that and might cause issues.

I was going back and forth on this. Attaching a bundle definition to a new claim was the easiest way to maintain the current interface for creating a driver operation.

I've been looking at this and it appears that runCmd.claimName will be the empty string in cases where you have to "new up" a claim from the bundle. I think the namespace for claim names is global, so such claims, if persisted, would tend to clobber each other. This doesn't feel good.

But I could have also implemented this with a temporary claim that served the purpose of creating the operation and then not persist the claim in duffle's store. This would maintain the constraint that claims should only be created on install. I ended up keeping the claim around because it was the only way for a user to keep track of which action had been run and its status.

Wouldn't it be better to change the contract and admit that there may not be a claim when a custom action is run against a bundle which is not currently installed?

The delete implications are valid concerns and I would be happy to make changes that didn't persist the claim in the duffle store. I think that's the right thing to do since the spec specifically defines claims as installed bundles. How would you feel about the PR if I made that change?

I think that would be an improvement, but it may not go far enough (see above).

Also, it seems slightly odd that modifying custom actions should save their claims, but non-modifying actions shouldn't as that makes the lifecycle of claims even more complicated to reason about.

This was just to maintain the current logic here.

Ok. It's not obvious to me that this logic needs to be preserved in this case.

I've been looking at this and it appears that runCmd.claimName will be the empty string in cases where you have to "new up" a claim from the bundle. I think the namespace for claim names is global, so such claims, if persisted, would tend to clobber each other. This doesn't feel good.

Right, so I think we're moving away from the idea of persisting it and we can use a randomly generated string for the temporary claim name. This would be sufficient for creating the driver operation and would it make it so that the claim name is only required if it already exists.

Wouldn't it be better to change the contract and admit that there may not be a claim when a custom action is run against a bundle which is not currently installed?

Can you expand on this? By admit, do you mean providing some log output that the claim does not exist and ask the user to retry with a bundle reference?

I've been looking at this and it appears that runCmd.claimName will be the empty string in cases where you have to "new up" a claim from the bundle. I think the namespace for claim names is global, so such claims, if persisted, would tend to clobber each other. This doesn't feel good.

Right, so I think we're moving away from the idea of persisting it and we can use a randomly generated string for the temporary claim name. This would be sufficient for creating the driver operation and would it make it so that the claim name is only required if it already exists.

Wouldn't it be better to change the contract and admit that there may not be a claim when a custom action is run against a bundle which is not currently installed?

Can you expand on this? By admit, do you mean providing some log output that the claim does not exist and ask the user to retry with a bundle reference?

No, I meant that we should admit the facts of the situation to the custom action which will then see an empty string as a claim name and understand that the bundle isn't installed and that there isn't a claim.
Specifically, I think we need to provide a way to create a driver operation when there isn't a claim.

Specifically, I think we need to provide a way to create a driver operation when there isn't a claim.

Yeah, that was the larger change I was trying to avoid. Especially since test coverage throughout the code base is inconsistent. I'll revisit this PR and see what it would take to make that change.


[Error: PineconeClient: Error calling upsert: Error: Vector dimension 1536 does not match the dimension of the index 1]

I ran the indexing script and returned this error
pnpm run ingest
[Error: PineconeClient: Error calling upsert: Error: Vector dimension 1536 does not match the dimension of the index 1]

Node.js v19.7.0
Waiting for the debugger to disconnect...
Waiting for the debugger to disconnect...

set number 1536
If the number decreases, try modifying Index Name


Need a flag to check if checking an expression for null or undefined has not that types

Often we have to check variables for empty values.
If variables has undefined or null types we check for that types.
But if we want to refactor our code, and replace nullable variables to always with value, we init its with empty values like false, 0, -1, ""
But checks that variables for null or undefined in other parts of program are still there, and make a lot of bugs
let a: string | undefined;
if (a !== undefined) {
   foo(a);
}

// and if we refactor

let a = '';
// In this check we potentially has a bug, cause undefined is not possible here,
if (a !== undefined) {
   foo(a);
}

https://palantir.github.io/tslint/rules/strict-type-predicates/ attempts this. It could use #13502 to work even better, but should be capable of handling undefined checks already.

Another issue linked with this one
// no error
if (typeof v === null) {}
if (typeof v === undefined) {}

typeof v  is always string, and check with raw null or undefined is incorrect

Covered by #14764?

@RyanCavanaugh yeah, it's the same what I want


Logging Pods are restarted- SCK version 1.4.1

What happened:
Pods were restarted multiple times with the below error
Exception encountered parsing namespace watch event. The connection might have been closed. Sleeping for 256 seconds and resetting the namespace watcher.Error while watching namespaces: too old resource version: 137665518 (137684936)
What you expected to happen:
No pod restarts
How to reproduce it (as minimally and precisely as possible):
The pods were automatically restarted after running for a while
Anything else we need to know?:
Environment:

Kubernetes version (use kubectl version):1.15.5
Ruby version (use ruby --version):
OS (e.g: cat /etc/os-release):Red Hat 7
Splunk version: 7.3.3
Others:

#413

Could you upgrade SCK to 1.4.3 or at least upgrade the logging pod image? splunk/fluentd-hec:1.2.4

Sure, I will upgrade the logging pod image and test it out

Hi @vbottu , closing this issue since there is no activity and it looks like upgrading logging pod image ,it got resolved for you.
You can re-open it if you face the same issue again . Thanks .


[Event Hubs] azure.eventhub.EventhubConsumerClient becomes exclusive receiver without checkpoint store

Package Name: azure-eventhub
Package Version: 5.1.0b1 and before
Operating System: Any
Python Version: Any

Describe the bug
azure.eventhub.EventHubConsumerClient.receive( ) and receive_batch( ) should not be exclusive receiver by default when there is no checkpoint store
To Reproduce

Use azure.eventhub.EventHubConsumerClient.receive() without a checkpointstore to receive from an event hub
Use the async counter part  azure.eventhub.aio.EventHubConsumerClient.receive() without a checkpoint and use the same consumer group to receive from the same event hub.

Expected behavior
They both receive concurrently.
But the 2nd receiver can't connect to a partition because the 1st receiver is exclusive and the 2nd isn't.

Fixed in release 5.1.0


support for better quality terminal graphics formats

for example, Sixels, Kitty, iTerm2, Unicode mosaics...
chafa https://hpjansson.org/chafa/ can handle generating these from normal images and has python bindings https://chafapy.mage.black/
talk about adding image support in textual: https://github.com/Textualize/textual/discussions/1191?sort=old

found this textual gui app with support for sixels and kitty graphics: https://github.com/mia-0/0x0?tab=readme-ov-file#moderation-ui


No way to end @example or intersperse multiple @example blocks

Presently ESDoc doesn't allow a way to end an @example block. One can create a second or follow up @example block, but no further normal blocks of documentation are possible. This makes long documentation blocks with multiple example blocks problematic. The worst case example I can show is from this private repo where I had to hack things together w/ bad formatting via using <code> blocks:
http://js.typhonrt.org/docs/backbone-parse-typhon-es6/class/backbone-parse-typhon-es6/src/BackboneQuery.js~BackboneQuery.html
As you can see one can use @example at the bottom of the block, but it's impossible to use @example multiple times interspersing it with other non-example documentation.
Suggestion: Provide a way to end an @example block; perhaps with multiple empty lines (2-3) or other directive.

@typhonrt I do not understand your comments... Please post your expecting code.
The following code is valid:
/**
 * @example
 * const str = 'this is first example';
 */
export class Foo1 {}

/**
 * @example
 * const str = 'this is first example';
 *
 * @example
 * const str = 'this is second example';
 */
export class Foo2 {}

And Manual Feature might help you.

This suggestion / feature is alleviated now that I know that markdown is supported.
The above example that you give is basic and works as expected. My comment was on inline @example usage when it may be something more like:
/**
 * @example
 * const str = 'this is first example';
 */
export class Foo1 {}

/**
 * Some description for class
 *
 * First example about some neat feature.
 * @example
 * const str = 'this is first example';
 *
 *
 * More text describing other features and the next example
 * @example
 * const str = 'this is second example';
 */
export class Foo2 {}

Currently ESDoc groups all examples at the bottom of the generated documentation block and doesn't allow interspersed @example blocks. It will include the inline text "More text describing other features and the next example" into the example blocks when it would be great for it to revert back to normal description block. This isn't so much of a problem as there is markdown support, but it would be nice to support inline examples as the syntax is just a bit nicer.
IE with markdown support the example above becomes:
/**
 * @example
 * const str = 'this is first example';
 */
export class Foo1 {}

/**
 * Some description for class
 *
 * First example about some neat feature.
 * **Example:**
 * ```
 * const str = 'this is first example';
 * ```
 *
 * More text describing other features and the next example
 * @example
 * const str = 'this is second example';
 */
export class Foo2 {}

Supporting inline or interspersed @example tags just allows a bit more syntactic sugar over having to use markdown for inline examples.

I understood your suggestion. However I do not apply the suggestion. Because ESDoc syntax will be too complex. Please use caption, markdown or manual integration.


Dictionary: Semantik of 'key'

While working with the Dictionary of V100 i ran into a couple of problems/questions around the result, especially the result.key.
In the first comments i will post these questions, so you can reply to every single question.
All questions are within this document:
2016-12-13_SymbologyQuestions.pdf
Contraints:
I retrieve the results of the dictionary using DictionarySymbolStyle::searchSymbol();
As parameter I used an empty SymbolStyleSearchParamter object, to get the complete model.
These results are retrieved using a widget-based executable.
The used mil2525d.stylx is from the installer of arcgisRuntmeV100, file size: 21468160Bytes
Key
The Dictionary returns properties containing

Name (String)
Tags (StringList)
SymbolClass (String)
Category (String)
Key (String)
The key has most of the time 8 digits: The SymbolSet (2 digits) and the Entity (6 digits).
But sometimes it´s different

Question 1: Key containing chars?
About 42 times the key does not contain digits but chars:
(Other problematic keys are: SN/A, SM/A, SL/A, SK/A, SJ/A, SD-/A
Is that intended or a bug?

Question 2: Key and it´s suffix “_C”
About 344 times i got a key containing the suffix “_C”:
(Ok, C is for Charlie. And Charlie could mean Mil2525Charlie.)

Ok, according to the standard this sidc is an area:

Why are there two results? What is the difference of these two results?

Question 3: Key and it´s suffix “_1”, “_2” ...
Often the key of 8 digits has a postfix like “_1”, “_3”, “_4”, “_6”.
First i assumed that this is the identity: “_1” for Unknown, “_3” for Friend and so on.
Here a screenshot my assumption is based on:

But after a while I found many entries not matching this assumption:

The last is “Neutral”, but the suffix is “_2”. But “_2” would be “Assumed Friend”
So, what is the meaning of the suffix?

Question 4: Key is empty
About 13 times the key is empty:

The 13 names of these entries are like

(1) Protections Line : Mine Cluster
(4) Protections Points : Nuclear…
(4) Maneuver Areas : Limited Access Area
(4) Fire Areas : Area Target

Is that intended or a bug?

Here is a quick/high-level answer:

Most of the key conventions were taken from the naming conventions of the source SVGs documented here:
https://github.com/Esri/joint-military-symbology-xml/tree/dev/svg
(These have the convention: 0 - Unknown,  1 - Friend,  2 - Neutral,  3 - Hostile)

However some of the data came from elsewhere, and also represented some of the same symbols, so has a different/exceptional naming convention
a. Tactical Graphics/Control Measures (Appendix B or Set 25) (Unknown = 1, Friend = 3, Neutral = 4,  Hostile = 6)
b. METOC (Appendix C, Set 45-47) _C
c. Important Note: these entries already exist in the items from 1 above - so you don't really need to present them in a user interface, but these items are the actual lines styles (versus just the simple point/ previews/examples from the standard)

The style also contains entries from 2525C/2525B - these are the alpha entries (this same style can be used to create those standards) - this naming convention is documented here

There are perhaps some old/unused entries in this style (ex: SN/A - which is an echelon indicator on a line) that we are waiting a few releases to make sure they are not needed before removing

If it helps, here is a query we used to filter entries not needed by the user in our user interface:
https://github.com/Esri/military-symbol-editor-addin-wpf/blob/master/source/ProSymbolEditor/ViewModels/MilitarySymbolDockpaneViewModel.cs#L1713

Hi Chris!
Thank you very much for your quick answer!
It´s very helpful.
Just one more related to Question 3: Key and it´s suffix “_1”, “_2” ...:
The first screenshot is showing 5 results:
a) one entry with key 25270100
b) four entries with key 25270100_x
From my point of view they are addressed be your note:

Important Note: these entries already exist in the items from 1 above - so you don't really need to present them in a user interface, but these items are the actual lines styles (versus just the simple point/ previews/examples from the standard)"

correct?
If so, which part will maybe removed in future?
Or in other words: Do you recommend to use a) or b)?

You can just use the (a) version - that is the style symbol part you need to get a symbol preview and get the tags/key needed to construct the symbol.
There are no plans to remove these particular entries, these other 4 entries are there:

in Control measures (set 25) to have different colored versions
in other symbols to have 4 versions that match exactly to the frame  (but note for these - there are only the _X versions)
but they should all have the same tags

Hi Chris!
Thanks a lot for your precise answer!


Use taskchain

https://github.com/aikar/TaskChain/
need a way to make that work with guice

the possibility to use it is there, closing


build: include verovio via python in docker image

This PR adds the Verovio Python Toolkit to the Docker image.
Leaving the Node implementation for the moment, in case something does not work. Node and scripts can then be removed in the future.

There is an issue with the installation of verovio via Python with arm64 architecture on Linux. I suggest to postpone the implementation of this PR after the release of v5.

Story of manual compilation for aarch64: https://marcin.juszkiewicz.com.pl/2021/03/25/from-a-diary-of-aarch64-porter-manylinux2014/


reified double element bug

special case, reduced from a much larger example
NOTE, in version 0.9.7 CPMpy gave no error but did not compute an answer either. With the cleaned up (soon landing) solver interfaces it does give an error (as it should).
Works properly with (non-flattened) minizinc solver, not with our flattening. See also a valid decomposition in the example
needs a bit more investigation...
# minimal? reproducing bug

import numpy as np
from cpmpy import *

# Constants for model
N = 6

model = Model()

# object type, or 0 (N of them)
otype = intvar(0,4,shape=(N),name="otype")

# subobjects, or -1 (N of them)
subobj = intvar(-1,N,shape=(N),name="subobj")

# of i is of type 4, then the subobj of i is also of type 4
model += [ (otype[i] == 4) == ( (otype[subobj[i]] != 4) ) for i in range(N)]

## properly handled by MiniZinc:
print(model.solve("minizinc"))
print("otype",otype.value())
print("subobj",subobj.value())

## a decomposition that is valid (and that we should do)
model2 = Model()
# of i is of type 4, then the subobj of i is also of type 4
model2 += [ (otype[i] == 4) == (
    any( [(subobj[i] == j) &
          (otype[j] != 4) for j in range(N)] )
                                                 ) for i in range(N)]
print(model2.solve())
print("otype",otype.value())
print("subobj",subobj.value())

## The error we get with our current flattening:
model.solve()

# the issue is that the domain of subobj (-1..N) is larger than the index of otype (0..N-1)
# so we can not simply use an element inside the outer '==' reification,
# we must only check the !=4 on the valid intersection of index values to determine truth of the reification

Our generic reification handling makes that there is no longer an error for the above.
However, the problem persists: we impose an element and that will force subobj to the domain of the array, but the reificatoin does not require that (e.g. subobj[.] = -1 should be valid).
At least, with the error gone it shows that the right place to do this right is in (the new) reify_rewrite, a solver that natively supports reified element would be aware of this, but it is our decomposition that is not.
# minimal? reproducing bug

import numpy as np
from cpmpy import *

# Constants for model
N = 3

model = Model()

# object type, or 0 (N of them)
otype = intvar(0,4,shape=(N),name="otype")

# subobjects, or -1 (N of them)
subobj = intvar(-1,N,shape=(N),name="subobj")

# of i is of type 4, then the subobj of i is also of type 4
model += [ (otype[i] == 4) == ( (otype[subobj[i]] != 4) ) for i in range(N)]

## properly handled by MiniZinc:
print(model.solve("minizinc"))
print("otype",otype.value())
print("subobj",subobj.value())

## a decomposition that is valid (and that we should do)
model2 = Model()
# of i is of type 4, then the subobj of i is also of type 4
model2 += [ (otype[i] == 4) == (
    any( [(subobj[i] == j) &
          (otype[j] != 4) for j in range(N)] )
                                                 ) for i in range(N)]
print(model2.solve())
print("otype",otype.value())
print("subobj",subobj.value())

# the issue is that the domain of subobj (-1..N) is larger than the index of otype (0..N-1)
# so we can not simply use an element inside the outer '==' reification,
# we must only check the !=4 on the valid intersection of index values to determine truth of the reification

# See: (model2 also has obobjs with value -1, model does not)
print("nModel ", model.solveAll()) #display=subobj))
print("nModel2", model2.solveAll())

still like before, but with a truly smaller testcase
iv = intvar(-1,3)
bv = boolvar()
arr = cpm_array([0,1,2])

e = (bv == (arr[iv] != 1))

print("Default: does not show iv=-1 or iv=3")
Model(e).solveAll(display=[iv,bv])

print("Correct with minizinc:")
Model(e).solveAll("minizinc",display=[iv,bv])

A correct rewriting would be:
e = (bv == any( [(iv == j) & (arr[j] != 1) for j in range(arr.shape[0])] ))

but isn't that the standard decomposition for an Element (equals value)?
In fact we currently don't implement one, because it requires to know the comparison... an Element for us is just the array access, not the comparison. This is important for other parts of the language, but for decompositions of 'numeric' global constraints it is actually... less clear


Configurable output format

This issue tracks adding a configurable output format engine within KraftKit and related commands which produce an informational output, for example kraft pkg ls or kraft pkg info.
Task list:

[ ] Investigate how kubectl get -o's output formatter works, it is quite configurable, e.g.:
-o, --output='': Output format. One of:
    json|yaml|wide|name|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=...

[ ] The implementation should migrate the use of the existing tool tableprinter and use instead a more relevant library for outputting formatted data.

[ ] A global option should be introduced into kraftkit.sh/config (Output) such that the format can be switched;

[ ] Log output format be adjusted to the desired format when its set to json, overriding log.type.

[ ] All tabular data should use a common interface such that its visible output can be adjusted dynamically and without requiring each instance to be reworked.

@nderjung I would like to work on it, Can you assign it to me :)

Hi @MdSahil-oss please feel free to tackle the issue! When ready, please submit a PR or feel free to reach out on Discord to discuss further.

How kubectl get -o is implemented in kubectl :- https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/get/get.go#L194

Upon implementation, tests should default to JSON output for better parsing. See https://github.com/unikraft/kraftkit/pull/405/files#r1180682845


STREAM: support sample rates below 2.5 MSps

Due to the limitations of the PLL in the fpga, the minimum supported sample rate is 2.5 MSps or 5 MHz PLL. Some applications (like GSM) and user testing may require lower sample rates.
One suggested fix may be to use a clock mux to switch between the PLL and an alternative clock for lower rates. A slower clock with appropriate phase relationship could be generated in register logic by oversampling the MCLK with a faster clock and creating the required phase delay.

Note: implemented in latest FPGA image, needs a software hook in the set PLL rate hook

SoDeRa gateware and firmware updated at 5e0fe51, when setting sampling rate below 2.5 Msps, sampling rate is switched to 0.541666 Msps

Does this mean that the sampling rate can be arbitrary above 2.5Msps and up to the upper limit, but below this it is fixed at 0.541666 Msps? Therefore, e.g. 2Msps, would not be possible?

Yes, that is correct

@guruofquality what would this mean for apps such as the BTLE demo that uses a 2M sample rate?
Would another option not be to have configurable decimation/interpolation in the FPGA and run the converters at higher rates (above 2.5 Msps)? Or would that be a lot of work or maybe require too much resource in the FPGA?

So I don't have a good grasp on what the PLLs are used for and what other technical barriers this Altera FPGA imposes.
I may have mentioned this before, but I can say that I solved this problem in the past on a series7 Zynq Xilinx FPGA doing the following: The RX MCLK clock is captured directly into a clock buffer and used to clock FPGA cores for LML capture with IDDR and for FIFOs in that clock domain. The TX MCLK is captured directly into a clock buffer and used to clock FPGA cores for LML generation with ODR and for FIFOs in that clock domain. There is however a single PLL used to generate the TX FCLK (feedback clock) which needs a 90 degree phase shift from the original TX MCLK. There is no RX PLL because there is no RX feedback clock required for the LML interface in this case.
So, when the TX MCLK sample rate becomes too low for the PLL to handle, the TX FCLK is swapped out with a clock mux for an oversampled clock. To generate the oversampled clock, the TX MCLK is sampled with a faster clock (in the 10s of MHz range) and the transitions are detected to generate the same 90 degree phase shift.
So, I don't recommend decimation/interpolation in the FPGA if the above is a possibility. But, I strongly think that this board needs to have a full range of flexible sample rates available to the user. I'm not very familiar with Altera, but I am curious to hear if this is a technical impossibility for Altera clocking.


Cannot install fv3net env due to missing package

Recent build of prognostic run image and installation of fv3net env for unit tests failed because of a missing package, atomicwrites==1.4.0.
See: https://app.circleci.com/pipelines/github/ai2cm/fv3net/12330/workflows/d399a1e3-7de0-4cfd-aff7-5f9282d99a50/jobs/111063 or https://app.circleci.com/pipelines/github/ai2cm/fv3net/12330/workflows/d399a1e3-7de0-4cfd-aff7-5f9282d99a50/jobs/111065
Steps to reproduce the problem:

pip install atomicwrites==1.4.0

Pin a different version of atomicwrites, or wait until pypi sorts it out?

see https://github.com/untitaker/python-atomicwrites/issues/61


REST interface

A small REST-esque service to query and update db to accompany the nifty CLI. This may be quicker for common use, as the app can stay resident and not have to load each time. Also potentially easier for integration in different apps.

I'm finally on it!

Fixed in https://github.com/Clinical-Genomics/loqusdbapi 😄 . Please try it out @dnil !

This is only a GET api at the moment, should be sufficient for usage with Scout. Will add more complete API when need arises. Please open issues about it in the other repo


[BugFix] Fix rate limit of tablet deletion

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #8600
Problem Summary(Required) ：

If FE found that meta already deleted when handling tablet report(i.e. tabletMeta == null), we should also control the rate limit of tablet deletion task sent to BE.


Display starter project list for odo catalog list components

/kind user-story
User Story
As a user, I want to see the list of starter project that I can create when viewing the component list by calling odo catalog list components.
In https://github.com/openshift/odo/issues/3005#issuecomment-623913169, one of the things that has been discussed is showing the list of available starter projects of a given component when doing catalog list on components. This will help the user to know what starter project is available when they call odo create with the --starter flag.
Acceptance Criteria

[ ] When calling odo catalog list components, the output should show the Starter project column to display the list of starter projects:

$ odo catalog list components
Odo Devfile Components:
NAME                 DESCRIPTION                            STARTER PROJECTS                REGISTRY
java-maven           Upstream Maven and OpenJDK 11                                          DefaultDevfileRegistry
java-openliberty     Open Liberty microservice in Java      user-app                        DefaultDevfileRegistry
java-quarkus         Upstream Quarkus with Java+GraalVM     quarkus-ex                      DefaultDevfileRegistry
java-springboot      Spring Boot® using Java                springbootproject               DefaultDevfileRegistry
nodejs               Stack with NodeJS 12                   nodejs-starter, other-starter   DefaultDevfileRegistry

[ ] The starter project list show also show in the output of odo catalog list component -o json
[ ] odo catalog list component -o json isn't consistent with other json output so that needs to be resolved as well

Links

Related issue:
https://github.com/openshift/odo/issues/3005

/kind user-story
/area devfile

Ran into a slight issue with this:
Currently odo catalog list components only downloads the index.json from a registry and retrieves the metadata from it, it does not download each devfile. However the index.json does not contain any information about starter projects.
So, if we want to include the list of starter projects in the output of odo catalog list components, we would to either:

Download every devfile when running odo catalog list components and retrieve the list of starter projects
Store the list of starter projects for a devfile in the index.json and update the index.json generation tool accordingly.

I'm in favour of 2, as it allows to keep downloading only the index.json and prevents us from having to do a costly download of every devfile in all of the registries visible to odo.

Ran into a slight issue with this:
Currently odo catalog list components only downloads the index.json from a registry and retrieves the metadata from it, it does not download each devfile. However the index.json does not contain any information about starter projects.
So, if we want to include the list of starter projects in the output of odo catalog list components, we would to either:

Download every devfile when running odo catalog list components and retrieve the list of starter projects
Store the list of starter projects for a devfile in the index.json and update the index.json generation tool accordingly.

I'm in favour of 2, as it allows to keep downloading only the index.json and prevents us from having to do a costly download of every devfile in all of the registries visible to odo.

I'm in favour of 2, as it allows to keep downloading only the index.json and prevents us from having to do a costly download of every devfile in all of the registries visible to odo, any time we need to get the devfiles in a registry.

I would go with 2 as well.
Index.json in the registry would be a good fit for this.
And more importantly, we already have problems with doing too many requests to GitHub.

Okay, so this issue will deliver two sets of changes:

Update the index.json generator in github.com/odo-devfiles/registry to add the list of projects in each devfile to the index.json
Update odo to retrieve the projects from the index.json

This issue is blocked on https://github.com/openshift/odo/issues/3263 and https://github.com/odo-devfiles/registry/pull/29

Is this really needed considering odo catalog describe component <component name> already does this?

Is this really needed considering odo catalog describe component <component name> already does this?

I would say that we don't need this anymore. odo catalog describe component <component name> should be enough.
I don't see real use-case when I would need to see all starter projects from all devfile components in the registry.

I'm fine with that
@elsony Any concerns closing this out in favour of the functionality that already exists in odo catalog describe component <component name>?

The main concern that I have is the out of the box experience.  I consider the component creation with starter is part of the out of the box user experience since it is most likely one of the first things that a new odo user (first time) will do.  The current usage flow will be user does odo catalog list components to find out the list of components. If the user wants to create a component with the starter, chances are the user will just do --starter.  For components with only one starter, this will work well.  However, for components with none or multiple starters, the default --starter path will fail and the user will need to know that they have to do odo catalog describe component to find out the list of available starters.
Given that in the future, we are going to share the same devfile repo with Che, there will be a mix of stacks, some with no starter, some with one starter and some with multiple, e.g. liberty stack is planning to have multiple starters. Without having that starter list show on the catalog list by default, the user will have to try out which has a starter and which one does not. This doesn't translate into a good user experience.  If we are concerned with adding things to the index.json without a proper design, I am ok with not showing that on catalog list for the time being.  As part of this item, we just need to make sure we provide a good error message when the user uses --starter on a stack that has none or multiple starter project defined, e.g. something like there are multiple starters available on this stack, use odo catalog describe component <component name> to see the list of available starters.  We can come back to this later to properly design on it.
One more thing to note is catalog describe is supposed to provide more detailed info, e.g. including description of each starter, repo location of the starter that the user can find the readme on it, etc. Therefore, even if we have the starter list in the catalog list, the catalog describe will not be a duplication of function.

@elsony I understand the concern. However, as a user the proposed solution of showing the list of starter projects in the component list is not very useful to me as I have no idea what these starters might be. Personally, I'd list the components and then use describe on the ones I might be interested in. describe should then properly describe the offered starters.
Actually, from a user's perspective, a fully interactive flow would be better: you'd just type odo create and then get guided by the tool: which component do you want to use, which starter, which options, etc. As it stands, there are way too many options to remember or even properly discover, resulting in needing to create more commands to help users out. Currently odo is too focused on being consumed by tools, in my opinion, and not enough about being consumed by humans.

So, I'm wondering if we can close out this issue? If there's a strong need for starter projects in odo catalog list components in the future, I guess we can reopen this in the future
On the topic of interactive mode, odo create's interactive mode already supports multiple starter projects. I created a devfile with two starter projects in it and added it to my own devfile registry to test:

@girishramnani On the earlier concerns with odo catalog list components -o json, its output appears to be consistent with other command's json outputs, so I'm not sure if anything needs to be done there either.

@johnmcollier I am ok to close this one

Closing out, let's re-open this if there's a strong desire for this issue again in the future


feat: grouping notification by defined labels

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

64 of 66 (96.97%)  changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.2%) to 77.346%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

core/notification/service.go
49
51
96.08%

Files with Coverage Reduction
New Missed Lines
%

internal/api/v1beta1/alert.go
1
79.49%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
3421

Relevant Lines:
4423

💛  - Coveralls


DER length is '110' and cannot be more than 4 bytes

It works from currently logged-in user, but when I try to launch it under other user (for example scheduling) I get this strange error
$SftpSession = New-SFTPSession -ComputerName $SftpAddress -Credential $Credential -KeyFile $KeyPath
New-SFTPSession : DER length is '110' and cannot be more than 4 bytes.
At line:1 char:16
+ ... tpSession = New-SFTPSession -ComputerName $SftpAddress -Credential $C ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-SFTPSession], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,SSH.NewSftpSession

Something wrong with your keyfile
What format is it in ? The library assume it is DER

Removed bits (b): ssh-keygen -m PEM -t rsa
Now it works fine.


Hompage-Design Wireframe Screen shots  Attached

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue no.)

Type of change

[ x] New feature (non-breaking change which adds functionality)

Explain the Testing instructions
Test Configuration:

Operating system: Windows
Version:
Text-editors used: Vs Code

Checklist:

[x ] My code follows the style guidelines of this project
[ x] I have performed a self-review of my own code
[ x] I have commented my code, particularly in hard-to-understand areas
[x ] I have made corresponding changes to the documentation
[x ] My changes generate no new warnings

ATTACH SCREEN-SHOTS
**I've attached the wireframe of the homepage As Instructed and added the screen shorts int he homepage folder inside wp-content inside the present source code.
here are the screen shorts of my design**

regarding the footer I had certain doubts and wanted to know your preferences so when you let me know your thoughts I'll work on that part .
I
f you approve of this style and after you assign me with this task , i'll start working on the codeing part.

@SAHU-01 Some issues -

Reduce size of logo and "SFC Foundation" text in navbar. It's too big and attracts way too much attention, we want the hero section and call-to-action buttons to be of highest priority and hierarchy.

The hero section lacks hierarchy, is too textual (Keep copywriting concise) and doesn't present the CTA buttons. Along with keeping the CTA buttons in the navbar, present them in the hero section too as the main task here is to get more volunteers or donation for taking the initiative further.
For hierarchy, try contrasting font sizes and font weights.
Hero image looks good, try to use it in a more creative way.

Do design other sections relating to "Events" and testimonials too.

Some websites you can take inspiration from  -
https://www.gatesfoundation.org/
https://acumen.org/
https://www.charitywater.org/  etc.
look up charity or foundations websites for inspirations.

And next time, do tag a mentor or admin for faster response.

Ok, so if I have to prioritize the donate and volunteer option more I can dedicate the entire hero section to that stuff, because then it would be prominent enough , it will be similar to the charity waters website ! ,If you agree to that I'll make those changes.
and comeing to the logo should I also reduce the size of of the foundation name as well.
For the events part , I'll surely work on it
as for the footer I had raised some doubts , could you clarify them, so that I can include it in the new wireframe and this time i'll be shareing a Balsamiq link for wireframe and a prototype .
And one last query ,what are your thoughts on website theme like do want it to be something professional or something more of UI stuff kind .
/cc @1502shivam-singh @himanshu007-creator @AnkushSinghGandhi @praveenscience

Ok, so if I have to prioritize the donate and volunteer option more I can dedicate the entire hero section to that stuff, because then it would be prominent enough , it will be similar to the charity waters website ! ,If you agree to that I'll make those changes.
and comeing to the logo should I also reduce the size of of the foundation name as well.
For the events part , I'll surely work on it
as for the footer I had raised some doubts , could you clarify them, so that I can include it in the new wireframe and this time i'll be shareing a Balsamiq link for wireframe and a prototype .
And one last query ,what are your thoughts on website theme like do want it to be something professional or something more of UI stuff kind .
/cc @1502shivam-singh @himanshu007-creator @AnkushSinghGandhi @praveenscience

more UI kind of stuff, it should feel friendly(😀 )

ok got that !
by tommorow morning i'll share the new designs.

Well if I'm adding the volunteer ,donate feature in the hero section then for the existent hero section ,I'll put it in the homepage below with a similar style of carousel.

What do you think?
/cc @1502shivam-singh @himanshu007-creator @AnkushSinghGandhi @praveenscience

here is the link to the new wireframe
https://balsamiq.cloud/swd5p85/psncvs9
I'll make the final  prototype once you agree to the arrangement of features present in this wireframe ,
and for the hero sectiion I'd suggest we should add a short video or gif and keep a similar looking donate box.
and I've added a fat footer style here so that the recent events can show up there along with social icons,etc.
/cc @1502shivam-singh @himanshu007-creator @AnkushSinghGandhi @praveenscience

Wireframe looks good, @SAHU-01 .
And good suggestion of short video/GIF in Hero section. It needs visual design work. If we don't get some video from the foundation, images can also be utilized in hero section, but this is an afterthought for now.
Also, the child image in hero section can be utilized somewhere else, or if you have some better ideas.

new commits are made to the previous pull request now the website looks like this


fix(sql-mapper): add types for tx option on all entity hooks

I have also made fields and where options on delete hook optional, I believe they are not required. let me know if I should revert it.
Example usecase:
/// <reference path="./global.d.ts" />
import { FastifyInstance } from "fastify";

export default async function (app: FastifyInstance) {
  app.platformatic.addEntityHooks("entity", {
    save: async (originalSave, opts) => {
      return app.platformatic.db.tx(async (tx) => {
        // Do other things atomically

        return originalSave({
          ...opts,
          tx,
        });
      });
    },
  });
}

@mcollina sorry, I forgot to add the method that creates a transaction.

I should mention that I found tx method here
https://github.com/ForbesLindesay/atdatabases/blob/master/packages/shared/src/BaseConnectionPool.ts#L92

Can you remediate the DCO? (click on details)


Login with email address

Can I use email address to login rather than phone number?
Thanks

It would be really useful


Non-ASCII chars on oled screen

Non ASCII chars are displayed as ? on the oled screen.
Is it possible to show utf8 chars ? If not maybe map accentuated chars to ASCII (as is commonly done with url slugiffy functions) ?
Can you work on it ?

Hi,
the font comes from

Do I understand it well: text to draw is in self.text ? So a quick fix could be to change accentuated chars to their counter parts ? (é as e, à as a) etc. Am I right ?

That would be one way to fix it, another would be to simply replace the font with an iso_8859_1 variant. This would probably be the easiest fix for you but not something I can merge into the master branch.


ci: propagate playground errors to cypress

the overall design of Cypress is very adverse to this idea, that you should be able to see what went wrong
so after a day's experimentation, here's roughly how we might be able to do it:

export the great big async code into a global promise
a. we have to configure webpack to do that
b. but webpack-dev-server breaks that, making all exports undefined. thanks https://github.com/webpack/webpack-dev-server/issues/2484
c. but webpack-dev-server 4 beta fixes that, so upgrade to that
d. but it also changes the configuration schema in ways that aren't documented yet
e. so fix those or move things into defaults and get rid of the configuration altogether. leaning towards the latter
reach in from cypress and grab that promise

so we won't need to do that hacky 'cypress-done-core' stuff anymore either
fixes #41

currently have to figure out all the timeout systems

.playground should be present really quickly
it could take a long time to fulfill
if it rejects, we should fail right away. but unfortunately cypress can't fathom the idea that the test makes progress other than towards success

ready for review
sample failure, which actually happened when I resolved a merge conflict wrong: https://github.com/oasisprotocol/oasis-sdk/runs/2223415566?check_suite_focus=true
  1) playground
       finishes:
     Error: bad secret key size
      at Function.nacl.sign.keyPair.fromSecretKey (webpack://playground/../../core/node_modules/tweetnacl/nacl-fast.js?:2320:11)
      at Function.fromSecret (webpack://playground/../../core/dist/signature.js?:101:39)
      at eval (webpack://playground/./src/index.js?:105:95)


[Bug]: Log the whole context information when StorageWrite API rejects the append requests.

What happened?
In the current implementation only the first element of the context gets considered to predicate if the rows should be forward propagated to the next PTransform, and that same context is used to print a warn log with the information of what happened.
Previously the full context was used to print the warn information providing full information on the ingestion error (quota related messages or details on internal errors) which now are lost complicating the troubleshooting of a pipeline.
Issue Priority
Priority: 2
Issue Component
Component: io-java-gcp

.take-issue


I can't change the message to the victim please help

I don't know if it's me or anyone else has this problem but i can't change the message

You can change the message in the code (line 7)

Yep, like @tiger0132 said, you should be able to change the message here: https://github.com/snollygolly/sourceio-automation/blob/master/main.js#L7
If that doesn't work, feel free to re-open the issue.


bazel remote: Server terminated abruptly (error code: 14, error message: 'Socket closed'...)

We're struggling with the following issue.
Versions:

Bazel client v6.1.1
Bazel remote v2.4.1 as docker container

Status:

we're currently migrating our project to bazel with bazel-remote (as docker image)
this migration went quite smooth till we tried to build Python for several linux distributions in parallel
as soon as we're using the bazel-remote cache, our builds may get terminated randomly with the following log:

[2023-06-14T12:58:08.436Z] # Execution platform: @local_config_platform//:host
[2023-06-14T12:58:08.436Z] [3 / 4] [Prepa] Foreign Cc - Configure: Building python
[2023-06-14T12:58:09.362Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/python_foreign_cc/Configure.log, 256.0 KiB / 2.9 MiB; 0s remote-cache
[2023-06-14T12:58:10.340Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/python_foreign_cc/Configure.log, 1.9 MiB / 2.9 MiB; 1s remote-cache
[2023-06-14T12:58:11.291Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 11.0 MiB / 22.1 MiB; 2s remote-cache
[2023-06-14T12:58:12.276Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 14.8 MiB / 22.1 MiB; 3s remote-cache
[2023-06-14T12:58:13.676Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 17.8 MiB / 22.1 MiB; 5s remote-cache
[2023-06-14T12:58:14.675Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 20.6 MiB / 22.1 MiB; 6s remote-cache
[2023-06-14T12:58:15.602Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 21.2 MiB / 22.1 MiB; 7s remote-cache
[2023-06-14T12:58:16.528Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 21.5 MiB / 22.1 MiB; 8s remote-cache
[2023-06-14T12:58:17.454Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 21.8 MiB / 22.1 MiB; 9s remote-cache
[2023-06-14T12:58:18.847Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 21.8 MiB / 22.1 MiB; 10s remote-cache
[2023-06-14T12:58:19.899Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 21.9 MiB / 22.1 MiB; 11s remote-cache
[2023-06-14T12:58:21.818Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 22.1 MiB / 22.1 MiB; 13s remote-cache
[2023-06-14T12:58:22.744Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 22.1 MiB / 22.1 MiB; 14s remote-cache
[2023-06-14T12:58:23.670Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 22.1 MiB / 22.1 MiB; 15s remote-cache
[2023-06-14T12:58:24.596Z] [3 / 4] Foreign Cc - Configure: Building python; Downloading external/python/copy_python/python/lib/libpython3.11.so, 22.1 MiB / 22.1 MiB; 16s remote-cache
[2023-06-14T12:58:26.147Z]
[2023-06-14T12:58:26.147Z] Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/home/jenkins/.cache/bazel/_bazel_jenkins/4a9333bbd1de8b96e8cb1132fb9c8ed1/server/jvm.out')

at the beginning of our quest, we realized that the bazel-remote gets killed by oom as we only had 8GB for the bazel remote server
as a mitigation we increased the RAM to 32GB and this limit is currently not reached anymore (peaks up to 10GB during job runs)
in the remote logs, we see things like:

GRPC BYTESTREAM READ FAILED TO SEND RESPONSE: 14c92d11f7e53a1d315e9125458a68105097d152dbee27cd063c9f6664c7453c rpc error: code = Unknown desc = connection error: desc = \"transport is closing\"\n","stream":"stdout","time":"2023-06-14T13:51:51.964021088Z"}

we tried to get more insights via pprof (on bazel-remote) and strace (on client side) but without success so far
with pprof, we tried getting 10s trace intervals - those interval were higher in the time a "crash" occurred
we can reproduce the issue by starting multiple bazel builds in parallel

Do you have any hints in which direction we could investigate further?

So we finally found the issue, it was the ulimit in our docker container: 1024 is way too low, we're now going with 16384:32768.
There are several issues around with bazel + ulimit but neither the logs nor the exception really pointed us in that direction. It was more by accident that we stumbled upon that docker config in our buildscripts.
Anyway, thanks to @ulrfa we had again a closer look on the client side and not anymore onto the bazel-remote!


[bounty] allow user to tune the FPS in timeline

definition of done:

user can select the frequency of frames in the timeline and it's saved in settings

/bounty 80
context:
i think it's hardcoded atm
https://github.com/mediar-ai/screenpipe/blob/b5a8bb6b3827620f57ab97ca94959e4c98b543d6/screenpipe-server/src/video_cache.rs#L658
i suggest you make it possible to pass the FPS in the API request and do some stuff with UI side (maybe a slider? or any better experience like a pinch or something? although assusming not everyone using a touchpad)

/attempt #693

/attempt #693

@louis030195 i kinda completed this issue, & added a slider in recorder settings


getting 403 when using Add-PnPSiteCollectionAdmin

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ba2dc727-d438-3510-19c1-f4c05580afcb
Version Independent ID: 8ea210be-0cb9-99a2-fd56-edd14a861d00
Content: Add-PnPSiteCollectionAdmin (PnP.Powershell)
Content Source: sharepoint/sharepoint-ps/sharepoint-pnp/Add-PnPSiteCollectionAdmin.md
Service: sharepoint-powershell
GitHub Login: @pamgreen-msft
Microsoft Alias: pamgreen

PS C:\WINDOWS\system32> Add-PnPSiteCollectionAdmin -Owners<EMAIL_ADDRESS>Add-PnPSiteCollectionAdmin : The remote server returned an error: (403) Forbidden.
At line:1 char:1

Add-PnPSiteCollectionAdmin -Owners<EMAIL_ADDRESS>
  + CategoryInfo          : WriteError: (:) [Add-PnPSiteCollectionAdmin], WebException
  + FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.Site.AddSiteCollectionAdmin

Hello @vayinapurapu
how are you?
Even though the PnP documentation articles are on this repository, I need to kindly redirect you to open PnP cmdlets issues/pull requests here, in that repository, where a member of the PnP community will be able to correct the article/review your PR with your feedback.
The documentation for PnP cmdlets is then synced from that repo into this one.
Thank you!

@get-itips Thank you very much for the contribution and sharing this explanation. @vayinapurapu Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


feat: update update form for Valibot v0.31.0

#1849

❓ Type of change

[x] 📖 Documentation (updates to the documentation or readme)
[ ] 🐞 Bug fix (a non-breaking change that fixes an issue)
[x] 👌 Enhancement (improving an existing functionality)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[x] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
Support for the Valibot v0.31.1 library rewrite.
This will make schemas from versions below v0.31.0 not work anymore.
📝 Checklist

[x] I have linked an issue or discussion.
[x] I have updated the documentation accordingly.

While working on this, @fabian-hiller also created his PR (#1848) but missed it. I'll close this in favor of his

Thanks for your efforts! 🙏


Try to replace JCEKS files by a PKCS#12

We are using JCEKS file to store some critical properties (ex: Hive MS db password).
We should try to use PKCS#12 files which are the new standard and easily manageable via Ansible.

Seems that this won't be possible...
The Hadoop documentation only mentions JCEKS files.
I tried with a PCKS12 file but it gave me a clear Caused by: java.io.IOException: Invalid keystore format error.
And we cannot go with JKS since this format apparently can not store entries of the SecretKeyEntry format:
Problem importing entry for alias javax.jdo.option.connectionpassword: java.security.KeyStoreException: Cannot store non-PrivateKeys.

It's also not possible to show the SecretKeyEntry with keytool or even with openssl after converting it to P12 according to this.

I thought we wanted to replace the JKS (java keystores / truststores) with PCKS#12, and not the JCEKS ?

I thought we wanted to replace the JKS (java keystores / truststores) with PCKS#12, and not the JCEKS ?

Yes but the idea was also to find a replacement for JCEKS in order to solve this issue cleanly: https://github.com/TOSIT-IO/tdp-collection/issues/180
But I guess we won't have any other choice than recreate JCEKS files with every config run.

Yes, but JCEKS are hadoop specific and cannot be replaced with a pcks#12 file.
JCEKS are only used for passwords and not truststores / keystores because it's meant to implement the Credential Provider api

Like @gboutry said, I think this issue cannot be done. For #180 we can make an Ansible module for updating JCEKS when needing with idempotence.

I agree that it's not possible, like I said in my first comment


Star Debugging

We have written the needed data into your clipboard because it was too large to send. Please paste.
Issue Type: Bug
I can't debug. I have a Gateway 64 bit laptop and OS Windows 7 Ultimate 64 bit. In the past it worked fine with MinGw64. Now, again I reinstalled it again with the MSYS2, as its documentation says. Starting debugging (F5) starts fine, but then doesn't follow to the debug console or external terminal (cmd). The path is correct: I check the versions of g++ and gdb and everything is fine.
VS Code version: Code 1.67.1 (da15b6fd3ef856477bf6f4fb29ba1b7af717770d, 2022-05-06T12:37:03.389Z)
OS version: Windows_NT x64 6.1.7601
Restricted Mode: No

System Info

Item
Value

CPUs
AMD Athlon(tm) II P320 Dual-Core Processor (2 x 2094)

GPU Status
2d_canvas: unavailable_softwarecanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: disabled_softwaremultiple_raster_threads: disabled_offoop_rasterization: disabled_offopengl: disabled_offrasterization: disabled_softwareraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: disabled_softwarevideo_encode: disabled_softwarevulkan: disabled_offwebgl: unavailable_softwarewebgl2: unavailable_software

Load (avg)
undefined

Memory (System)
1.75GB (0.42GB free)

Process Argv
--crash-reporter-id ddb4a07f-fe9a-496d-aeb4-0cddb22bfdd6

Screen Reader
no

VM
0%

Extensions (6)

Extension
Author (truncated)
Version

better-cpp-syntax
jef
1.15.13

vscode-language-pack-es
MS-
1.67.3

cpptools
ms-
1.10.2

cpptools-extension-pack
ms-
1.2.0

printcode
nob
3.0.0

vscode-icons
vsc
11.11.0

(1 theme extensions excluded)

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyl392:30443607
pythontb:30283811
pythonvspyt551:30345470
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805cf:30301675
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30449410
pythonvsnew555:30457759
vscscmwlcmt:30465135
cppdebug:30475216
vsclangdc:30486549

Please file this on the repo of the C++ extension


Add Openstack Swift API Support

https://docs.openstack.org/api-ref/object-store/
Developers can run tests using https://hub.docker.com/r/openstackswift/saio

Sure, I will take a look into this implementation.

Openstack Swift API support has been added, let's add it's behavior test!


Feature/issue 202 vectorize all

Submisison Checklist

[x ] Run unit tests: ./runTests.py test/unit
[ x] Run cpplint: make cpplint
[ x] Declare copyright holder and open-source license: see below

Summary:
This pull request is to add vectorized function to stan/math/prim/mat.hpp so they can be vectorized in Stan.
Intended Effect:
None to users. Internally, we add vectorized functions to stan/math/prim/mat.hpp.
How to Verify:
Check that vectorized functions are added correctly and tests still pass.
Side Effects:
None.
Documentation:
None needed.
Reviewer Suggestions:
Copyright and Licensing
Per Michigan's copyright laws, Rayleigh Lei is the copyright holder.
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

I couldn't add lgamma or fabs because there are other template functions using those functions and unit tests fail when adding them to stan/math/prim/scal/fun/mat.hpp. Specifically, stan/math/prim/scal/fun/lbeta.hpp is templated and uses lgamma whereas stan/math/prim/scal/fun/grad_2F1.hpp is templated and uses fabs.

I'm working on this in #370 starting by eliminating math.h branch and then will add vectorization for what's left in #347.  #370 already has a pull request and #347 has already added a few more functions that weren't in the original request, but I'm going to stage #347 after #370.

I wasn't sure how this pull request was supposed to help with only defining a mat.hpp header file.

I'm confused too. Do those headers all exist?

Got it. I'll change the description, but I was adding the functions already vectorized in #202. In that huge pull request, I forgot to add those functions to stan/math/prim/mat.hpp. As a result, I think only log and exp can be vectorized in Stan.

I was looking at this and realized that I hadn't added the headers to stan/math/prim/mat.hpp in the last pull request. My other comment was about functions that are vectorized in stan/math, but couldn't be added to stan/math/prim/mat.hpp without the unit tests failing. I did some investigation and my comment is my guess at why I couldn't add them without the tests failing.

When I tried adding the function signature test for the matrix form of abs in Stan, I get make error 252. However, the test is a copy of the exp matrix test that passes, only with abs replacing the exp calls. Based on this, I think the cause of the make error is because I hadn't added the appropriate headers to stan/math/prim/mat.hpp in the last vectorize-all pull request in stan-math. Hence, I created this pull request to fix that problem by adding the headers to stan/math/prim/mat.hpp.
Indeed, I just tried adding the function signature test for the matrix form of log in Stan because the header file is in stan/math/prim/mat.hpp. The log matrix test is a copy of the exp matrix test, again with log replacing the exp calls. The log test did not error out.
So, if I can get this pull request in, minus the two functions I have had trouble adding to stan/math/prim/mat.hpp, I should be able to push the vectorized functions up to Stan.

When my pull request for remving math.h is merged (should be soon), then we will have stan::math::round we can use instead of ::round from math.h, which isn't portable to Windows compilers.

Why are you removing includes like cov_exp_quad.hpp?

Actually, that's already merged, so you should update and use stan::math::round.

I'll switch it. I'm using ::round because the integration tests were failing due to std::round last week.

No functions were removed. I essentially took out all functions in stan/math/prim/mat/fun/, added vectorized function headers, sorted all functions alphabetically using bash's sort, and then put it back in stan/math/prim/mat.hpp. As a result, some existing functions got moved around.

(1) Thanks.
You only get std::round in C++11.   But it's built into some
of the g++ and clang++ implementations.  Overall, the libraries
are a mess before C++11 but get standardized properly in C++11.
(2) I see.  That's OK.  They should go in the alphabetical
order you rearranged them into.

Bob

On Oct 4, 2016, at 12:32 PM, rayleigh<EMAIL_ADDRESS>wrote:

I'll switch it. I'm using ::round because the integration tests were failing due to std::round last week.

No functions were removed. I essentially took out all functions in stan/math/prim/mat/fun/, added vectorized function headers, sorted all functions alphabetically using bash's sort, and then put it back in stan/math/prim/mat.hpp. As a result, some existing functions got moved around.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I updated round to use stan::math::round. Is this good to merge? Or, do I need to also add lgamma and fabs to stan/math/prim/mat.hpp as well? Or, is there something else as well?

I'm OK with this merging---I can fix the merge conflict it'll
cause on my existing branch that vectorizes the remaining
functions.  I'm stuck on translation unit tests failing.
But what I don't see is how you can get away with putting
the code in stan/math/prim/mat.hpp---I wound up needing to
put it in stan/math/mix/mat.hpp to make sure it went after
all the core fwd and rev stuff so it could get instantiated
properly.

Bob

I'm curious what you mean by that.
If I include the vectorized version of lgamma.hpp in prim/math/mat.hpp and run test/unit/math/mix/scal/meta/child_type_test.cpp, I get the following error messages:
In file included from test/unit/math/mix/scal/meta/child_type_test.cpp:3:
In file included from ./test/unit/math/prim/scal/fun/promote_type_test_util.hpp:4:
In file included from ./stan/math/prim/mat.hpp:48:
In file included from ./stan/math/prim/mat/fun/abs.hpp:4:
./stan/math/prim/mat/vectorize/apply_scalar_unary.hpp:39:41: error: implicit
      instantiation of undefined template
      'Eigen::internal::traits<unsigned int>'
      typedef typename Eigen::internal::traits<T>::Scalar scalar_t;
                                        ^
./stan/math/prim/mat/fun/lgamma.hpp:33:21: note: in instantiation of template
      class 'stan::math::apply_scalar_unary<stan::math::lgamma_fun, unsigned
      int>' requested here
    inline typename apply_scalar_unary<lgamma_fun, T>::return_t
                    ^
./stan/math/prim/mat/prob/lkj_corr_log.hpp:65:52: note: while substituting
      deduced template arguments into function template 'lgamma'
      [with T = unsigned int]
            - 0.25 * (Km1 * Km1) * LOG_TWO - Km1 * lgamma((K + 1) / 2);
                                                   ^
lib/eigen_3.2.9/Eigen/src/Core/util/ForwardDeclarations.h:17:29: note: template
      is declared here
template<typename T> struct traits;
                            ^
In file included from test/unit/math/mix/scal/meta/child_type_test.cpp:3:
In file included from ./test/unit/math/prim/scal/fun/promote_type_test_util.hpp:4:
In file included from ./stan/math/prim/mat.hpp:48:
In file included from ./stan/math/prim/mat/fun/abs.hpp:4:
./stan/math/prim/mat/vectorize/apply_scalar_unary.hpp:45:39: error: type
      'unsigned int' cannot be used prior to '::' because it has no members
      typedef Eigen::Matrix<scalar_t, T::RowsAtCompileTime,
                                      ^
./stan/math/prim/mat/vectorize/apply_scalar_unary.hpp:39:41: error: implicit
      instantiation of undefined template
      'Eigen::internal::traits<stan::math::var>'
      typedef typename Eigen::internal::traits<T>::Scalar scalar_t;
                                        ^
./stan/math/prim/mat/fun/lgamma.hpp:33:21: note: in instantiation of template
      class 'stan::math::apply_scalar_unary<stan::math::lgamma_fun,
      stan::math::var>' requested here
    inline typename apply_scalar_unary<lgamma_fun, T>::return_t
                    ^
./stan/math/prim/scal/fun/lbeta.hpp:61:14: note: while substituting deduced
      template arguments into function template 'lgamma' [with T =
      stan::math::var]
      return lgamma(a)
             ^
./stan/math/rev/scal/fun/grad_inc_beta.hpp:37:20: note: in instantiation of
      function template specialization 'stan::math::lbeta<stan::math::var,
      stan::math::var>' requested here
      var c3 = exp(lbeta(a, b)) * inc_beta(a, b, z);
                   ^
lib/eigen_3.2.9/Eigen/src/Core/util/ForwardDeclarations.h:17:29: note: template
      is declared here
template<typename T> struct traits;
                            ^
In file included from test/unit/math/mix/scal/meta/child_type_test.cpp:3:
In file included from ./test/unit/math/prim/scal/fun/promote_type_test_util.hpp:4:
In file included from ./stan/math/prim/mat.hpp:48:
In file included from ./stan/math/prim/mat/fun/abs.hpp:4:
./stan/math/prim/mat/vectorize/apply_scalar_unary.hpp:45:42: error: no member
      named 'RowsAtCompileTime' in 'stan::math::var'
      typedef Eigen::Matrix<scalar_t, T::RowsAtCompileTime,
                                      ~~~^
4 errors generated.
make: *** [test/unit/math/mix/scal/meta/child_type_test.o] Error 1
make test/unit/math/mix/scal/meta/child_type_test failed
exit now (10/11/16 13:34:36 EDT)

If I go into stan/math/prim/mat/prob/lkj_corr_log.hpp and add using stan::math::lgamma to the function do_lkj_constant, the only function that uses lgamma, I get the following error message from running the same test:
In file included from test/unit/math/mix/scal/meta/child_type_test.cpp:3:
In file included from ./test/unit/math/prim/scal/fun/promote_type_test_util.hpp:4:
In file included from ./stan/math/prim/mat.hpp:48:
In file included from ./stan/math/prim/mat/fun/abs.hpp:4:
./stan/math/prim/mat/vectorize/apply_scalar_unary.hpp:39:41: error: implicit
      instantiation of undefined template
      'Eigen::internal::traits<stan::math::var>'
      typedef typename Eigen::internal::traits<T>::Scalar scalar_t;
                                        ^
./stan/math/prim/mat/fun/lgamma.hpp:33:21: note: in instantiation of template
      class 'stan::math::apply_scalar_unary<stan::math::lgamma_fun,
      stan::math::var>' requested here
    inline typename apply_scalar_unary<lgamma_fun, T>::return_t
                    ^
./stan/math/prim/scal/fun/lbeta.hpp:61:14: note: while substituting deduced
      template arguments into function template 'lgamma' [with T =
      stan::math::var]
      return lgamma(a)
             ^
./stan/math/rev/scal/fun/grad_inc_beta.hpp:37:20: note: in instantiation of
      function template specialization 'stan::math::lbeta<stan::math::var,
      stan::math::var>' requested here
      var c3 = exp(lbeta(a, b)) * inc_beta(a, b, z);
                   ^
lib/eigen_3.2.9/Eigen/src/Core/util/ForwardDeclarations.h:17:29: note: template
      is declared here
template<typename T> struct traits;
                            ^
In file included from test/unit/math/mix/scal/meta/child_type_test.cpp:3:
In file included from ./test/unit/math/prim/scal/fun/promote_type_test_util.hpp:4:
In file included from ./stan/math/prim/mat.hpp:48:
In file included from ./stan/math/prim/mat/fun/abs.hpp:4:
./stan/math/prim/mat/vectorize/apply_scalar_unary.hpp:45:42: error: no member
      named 'RowsAtCompileTime' in 'stan::math::var'
      typedef Eigen::Matrix<scalar_t, T::RowsAtCompileTime,
                                      ~~~^
2 errors generated.
make: *** [test/unit/math/mix/scal/meta/child_type_test.o] Error 1
make test/unit/math/mix/scal/meta/child_type_test failed
exit now (10/11/16 13:36:48 EDT)

If I look at stan/math/prim/scal/fun/lbeta.hpp, I see that the function is templated. My guess is that because of the includes, it's pulling in the unary vectorize code out of order.

The fundamental problems are:

Bringing in a template definition before all
the classes needed to instantiate it.  That's why I couldn't
figure out how to include in prim/mat.hpp.  But I see
what you did and am going to try following that pattern right now.

Ambiguities due to not having a most specific
implementation.  For example, providing an int
argument to a function defined only for double and
var --- int can be promoted to either one.  That's why I've
added explicit int definitions where necessary.

I have the basic functions all sorted in a pull request that's
just awaiting Daniel's help for the multiple translation units.
And I vectorized the rest of the functions in a pull request waiting
on the first pull request.
I'd suggest just working on a pull request to get the rest of
the functions you wrote vectorized in Stan and by the time you're
done, I should have the math library sorted.

Yup. I need to work on that multiple translation unit thing.

On Oct 11, 2016, at 2:30 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:
The fundamental problems are:

Bringing in a template definition before all
the classes needed to instantiate it. That's why I couldn't
figure out how to include in prim/mat.hpp. But I see
what you did and am going to try following that pattern right now.

Ambiguities due to not having a most specific
implementation. For example, providing an int
argument to a function defined only for double and
var --- int can be promoted to either one. That's why I've
added explicit int definitions where necessary.

I have the basic functions all sorted in a pull request that's
just awaiting Daniel's help for the multiple translation units.
And I vectorized the rest of the functions in a pull request waiting
on the first pull request.
I'd suggest just working on a pull request to get the rest of
the functions you wrote vectorized in Stan and by the time you're
done, I should have the math library sorted.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I see. Thanks for letting me know.

I'd suggest just working on a pull request to get the rest of the functions you wrote vectorized in >Stan and by the time you're done, I should have the math library sorted.
That sounds good. Could I get this pull request merged then? I currently get a make error 252 when I try to vectorize in Stan the functions that are vectorized in stan/math. This pull request is to add the functions to stan/math/prim/mat.hpp with the exception of lgamma and fabs for reasons we've discussed above. Doing so will allow me to vectorize them in Stan because I've only been able to vectorize in Stan log and exp, functions that are in stan/math/prim/mat.hpp.